Paper deep dive
From Profiling to Synthesis: Benchmarking Implicit Behavioral Alignment in Personalized LLM Agents
Jiajia Song, Bobo Li, Haiwen Yi, Zibo Ji, Meishan Zhang, Hao Fei, Min Zhang, Mong-Li Lee, Wynne Hsu
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large Language Models have enabled increasingly capable autonomous agents, yet personalization remains critical for making such agents practically useful. Recent benchmarks have begun evaluating personalization in agents, but they largely rely on static preference snapshots, fixed interaction logs, or question answering over predefined user profiles. Such designs fail to capture the complexity of evolving user preferences and neglect preference-conditioned task execution-a discrepancy we term as the knowledge-to-action gap. To address this challenge, we introduce IBA-Bench, a benchmark for implicit behavioral alignment constructed from longitudinal interaction histories that contain noise, implicit cues, and temporal inconsistencies. Unlike prior work, IBA-Bench evaluates whether an agent can execute tasks while satisfying implicit user constraints inferred from historical interactions. We further propose IBA-Agent, an agent framework that reconciles conflicting priorities through broad retrieval and trajectory-level alignment. Experiment results on IBA-Bench show that effective personalization remains a significant challenge for state-of-the-art LLM agents, and the proposed IBA-Agent substantially improves behavioral alignment in complex scenarios across nine application domains.
Tags
Links
- Source: https://arxiv.org/abs/2608.02171v1
- Canonical: https://arxiv.org/abs/2608.02171v1
Trouble viewing inline? Open PDF directly →
Full Text
78,716 characters extracted from source content.
Expand or collapse full text
From Profiling to Synthesis: Benchmarking Implicit Behavioral Alignment in Personalized LLM Agents Jiajia Song 1 Bobo Li 1 Haiwen Yi 2 Zibo Ji 3 Meishan Zhang 4 Hao Fei 5 Min Zhang 4 Mong-Li Lee 1 Wynne Hsu 1 1 National University of Singapore 2 University of Toronto 3 University of Minnesota Twin Cities 4 Harbin Institute of Technology, Shenzhen 5 University of Oxford jjiasong,libobo,dcsleeml,dcshsuw@nus.edu.sg, hao.fei@bdi.ox.ac.uk Abstract Large Language Models have enabled increas- ingly capable autonomous agents, yet personal- ization remains critical for making such agents practically useful. Recent benchmarks have be- gun evaluating personalization in agents, but they largely rely on static preference snapshots, fixed interaction logs, or question answering over predefined user profiles. Such designs fail to capture the complexity of evolving user preferences and neglect preference-conditioned task execution—a discrepancy we term as the knowledge-to-action gap. To address this chal- lenge, we introduce IBA-BENCH, a benchmark for implicit behavioral alignment constructed from longitudinal interaction histories that con- tain noise, implicit cues, and temporal incon- sistencies. Unlike prior work, IBA-BENCH evaluates whether an agent can execute tasks while satisfying implicit user constraints in- ferred from historical interactions. We further propose IBA-AGENT, an agent framework that reconciles conflicting priorities through broad retrieval and trajectory-level alignment. Ex- periment results on IBA-BENCH show that ef- fective personalization remains a significant challenge for state-of-the-art LLM agents, and the proposed IBA-AGENT substantially im- proves behavioral alignment in complex sce- narios across nine application domains. 1 Introduction Large Language Models (LLMs) (Achiam et al., 2023; Team, 2025; Guo et al., 2025) have evolved from conversational chatbots into autonomous agents capable of executing complex real-world tasks (Yang et al., 2018, 2024). This shift makes personalization essential as users now expect as- sistants that can adapt to their preferences and constraints rather than simply generic tool execu- tors (Samuel et al., 2025). Achieving such personal- ization is a challenge because user preferences are rarely explicit; instead, they dynamically evolve and remain implicitly embedded within lengthy interaction histories (Zhang et al., 2025). Captur- ing and synthesizing these dynamic signals across long interaction histories is needed for building the next-generation personalized agents. Recent works have introduced various person- alized agents and evaluation frameworks. Early efforts extracted static profiles for response genera- tion (Joko et al., 2024), collapsing preferences into flat tags and ignoring temporal dynamics. Subse- quent benchmarks incorporated raw dialogue his- tories to preserve realism. For instance, HiCU- PID (Mok et al., 2025) retains conversational tra- jectories, while PersonaBench (Tan et al., 2025) models dynamic attribute evolution. Task-oriented evaluations like PersonaLens (Zhao et al., 2025) replace interaction histories with summarized ab- stracts. However, all these evaluations predomi- nantly rely on extraction-based Question Answer- ing (QA). As a result, they fail to adequately mea- sure an agent’s ability to integrate implicit, evolv- ing user preferences into complex actions and deci- sions. We refer to this limitation as the knowledge- to-action gap. This gap highlights a stark discon- nect: an agent may possess crucial knowledge yet fail to apply it during task execution. Consider a user with a historical preference for spicy food. Recent interaction history reveals that the user has undergone a wisdom tooth extraction (Figure 1). When asked to order dinner, current agents typically order a spicy meal based on the static user preference profile. Yet, the same agent is able to recall that the user’s tooth extraction in- dicating that it possesses the knowledge. Since the dental event and dining request seems unrelated, agents frequently overlook this connection unless explicitly prompted. An ideal personalized agent would implicitly synthesize these signals and proac- tively recommend soft food. Bridging this gap re- quires moving beyond static profiling to dynamic synthesis, where agents infer implicit constraints and reconcile conflicts autonomously. 1 arXiv:2608.02171v1 [cs.AI] 3 Aug 2026 More personalized task completion Food | Preference | Dynamic states 1USER: I’m hungry. Order me food？ User Profile Log Flavor Spicy Role Student Age 24 Major Physics Ordinary Agent Only apply: “Likes spicy food” Food | Preference Context Fusion Profile + Dynamic Event Dynamic Event Recent tooth extraction 2AGENT: Sure! I know you like spicy food. I’l order something spicy for you. 3 USER: Wait, what? Didn’t you notice I’m not feeling wellthese days? 4 AGENT: Oh, wait... you just had a wisdom tooth extraction. Sure! Although you like spicy food, since you recentlyhad a tooth extraction, I’l order some softfood for you. 1USER: Please help me order some food. Thanks, that’s just what I need. IBA-Agent Figure 1: Personalization beyond QA: knowledge-to-action gap between preference inference and task execution. In this work, we propose IBA-BENCH, a bench- mark specifically designed to assess implicit be- havioral alignment in agents. The primary objec- tive of IBA-BENCH is to evaluate and bridge the knowledge-to-action gap. At the data level, we construct longitudinal interaction histories contain- ing implicit preference cues, dynamic attributes, and conflicts between long-term traits and short- term states. At the task level, we move beyond explicit QA and simple preference matching by providing agents with full interaction records and the user request. The agent carries out the request and we evaluate whether the request is successfully executed and that the implicit constraints inferred from the interaction history are satisfied. We also introduce IBA-Agent, a personalized agent framework to bridge the gap between histori- cal preference understanding and active task execu- tion. IBA-Agent consists of two key components: (i) Deep Retrieval, which isolates relevant behav- ioral cues from noisy interaction histories, and (i) Broad Thinking and Deep Alignment, which syn- thesizes these signals into actionable constraints and planning logic prior to generation. Extensive experiments on IBA-BENCH reveal that standalone LLMs (Achiam et al., 2023; Guo et al., 2025) and even strong general-purpose agents such as Claude Code and Hermes Agent struggle significantly on these synthesis-heavy ex- ecution tasks. These results empirically validate the knowledge-to-action gap. In contrast, IBA- Agent substantially improves behavioral alignment in complex scenarios, demonstrating the effective- ness of explicit trajectory-level synthesis. Our main contributions are: •We advocate a shift in personalized agent re- search from static preference profiling to dy- namic preference synthesis, and identify the knowledge-to-action gap as a key bottleneck in translating inferred knowledge into effec- tive task execution. • We introduce IBA-BENCH, a benchmark that evaluates implicit behavioral alignment through concrete task execution in realistic set- tings characterized by dynamic, conflicting, and noisy contexts. •We propose IBA-Agent, a personalized agent framework that combines deep retrieval with trajectory-level synthesis, providing a strong empirical baseline for building agents capable of personalized reasoning. 2 Related Work LLM-Based Agents have emerged as an active research area with the rapid advancement of LLMs’ reasoning (Wei et al., 2022), planning (Valmeekam et al., 2023), and tool use (Schick et al., 2023; Li et al., 2026). Compared to generic agents, these agents seek to operate in dynamic and diverse inter- action scenarios by aligning decision-making with user-specific characteristics (Wang et al., 2024). Current approaches typically introduce personal- ization through explicit user modeling (Zhang et al., 2025; Afzoon et al., 2024), memory or retrieval- augmented mechanisms (Ram et al., 2023), and adaptation based on user feedback (Stiennon et al., 2020) with the goal of capturing both long-term user preferences and evolving interaction con- texts (Westhäußer et al., 2025; Park et al., 2023). Existing research remains largely method-centric, focusing on personalization mechanisms and model design rather than systematic evaluation of per- sonalization capabilities (Mohammadi et al., 2025; Hao et al., 2025). 2 Construct profile Example persona: Marcus Tan Age: 41 |Male| Worker Personality: methodical systems- oriented problem solver. Emotional State: Steady,cautious Stressors、ldentity、Major...... 400 seed personas nLong-term(stable) nShort-term (vary) Background attributes Personality Traits Transient states Current role Current task New task under the same scenario Help me write a blog post. Write a blog post analyzing the challenges and rewards of starting a specialized niche blog from scratch...... Agent output action space Generate (text) e.g., write the blog post CallAPI(params) e.g., order food Behavioral Evaluator Assess behavioral correctness (preference-specific criteria) Task Instantiation&Validation Score Behavioral successrate Write a blog post reflecting on switching from a 9-5 office job to remote freelancing ...... Make sure to write it in the first person. From Office Badge to Browser Tabs: How Leaving 9-5 Reshaped My Work and Life. When I left my traditional 9-5 office job for remote freelancing. 2025-06-23 05:01:00 Title: From Office Badge to Browser Tabs: What l Learned Moving from 9-5 to Remote Freelancing..... Generate Interaction History Example interaction history Distractors (task-irrelevant chat) Casual small talk EntertainmentLogistics Input: persona + hidden preferences + task contexts (only the generator can see hidden preferences) Multi-agent Generator Figure 2: Benchmark construction flow chart. Benchmarks for Personalized Agents are derived from general-purpose evaluation settings (Liu et al., 2024b), including QA, tool use, and short- horizon planning tasks (Hu and Shu, 2023). These benchmarks measure task success or accuracy un- der static assumptions and task-level metrics (Zhu et al., 2025) . While effective for assessing gen- eral agent capabilities, they offer limited support for evaluating personalized behaviors (Hao et al., 2025). Existing personalization benchmarks op- erationalize personalization as profiling, either by extracting user attributes via QA or by aligning responses under explicit and static preference spec- ifications (Mok et al., 2025; Tan et al., 2025; Zhao et al., 2025). These settings under-specify the core challenge faced by agents: synthesis, which in- volves integrating implicit, potentially conflicting, and time-varying user signals into task execution over interaction histories. Motivated by these limi- tations, we introduce IBA-BENCH. A comparison with prior benchmarks is summarized in Table 1. BenchmarkHistory Dynamic Implicit Task Behavior HiCUPID✓✗✓✗ PersonaBench✓✗ PersonaLens✗✓✗ IBA-BENCH (Ours)✓ Table 1: Comparison of Persona-related Benchmarks. Our proposed IBA-BENCH is the only benchmark that covers all dimensions. 3 Benchmark Construction To bridge the gap between static profiling and dynamic synthesis, we construct IBA-BENCH through a multi-stage pipeline designed to simulate the noise, implicitness, and conflicts inherent in real-world personalization. An IBA-BENCH in- stance is defined by(H u , x, s,Y,E ), whereH u is the interaction history of useru,xis the current task query under scenarios,Yis the action space, andEis the behavioral evaluator. The action space Y includes two types: • Generate(text): produce personalized textual outputs such as writing or planning; • CallAPI(params) : generate structured parame- ters for tool-mediated actions such as ordering food or making a booking. GivenH u ,x, ands, the agent must infer the histor- ical signals most relevant to the current task and de- termine an action y ∈Y , where y = f θ (H u , x, s). The evaluatorEassesses the behavioral correct- ness of the final action using type-specific crite- ria such as generation quality and API-parameter matching. Detailed scoring definitions and exam- ples are provided in Appendix A.2. Figure 2 shows the construction process of IBA-BENCH. Construct Personas. We define 400 seed personas as the basic units for modeling user-specific vari- ation. Each persona is multi-dimensional and for- malized asp = (B, T, E, R), whereBdenotes background attributes,Tpersonality traits,Etran- sient states (e.g., stress or urgency), andRthe cur- rent professional or social role. Note thatBand Tare long-term attributes that remain relatively stable over time, whileEandRare short-term preferences that may change and contradict ear- lier behavior. This formulation allows a persona to capture personal traits, situational states, and professional or social roles, enabling coherent yet diverse behaviors across scenarios. Generate Interaction History. To mimic realistic interaction history, we use a multi-agent framework to generate interactions where informative prefer- 3 Domain Scale Stats# Evidence# Constraints SDPI12345+12345+ Writing and Design1655107502712678361686619928216043 Work1253171,15539438016511110530161338321305 Daily Consumption839151,086511290217363246182401276181 Planning1039161,15153232918242664120648938629 Exercise and Health4162078132524711341551154173252291 Transportation4141451828811711201216716919764 Medical Services311155242901131210018902311850 Leisure Activities391433313710462161412601481130 Information Management626176642932179429312373185161222 Overall66262156,9623,0412,0641,1493363722681,0922,4162,0511,135 Table 2: Statistics of domains in the benchmark. S: number of scenarios in each domain; D: total number of preference dimensions across scenarios; P: integer-truncated average number of preference values per scenario; I: number of task instances. The columns under #Evidence and #Constraints report the distributions of task instances by the number of associated evidence items and preference constraints in each instance, respectively. In both column groups, “5+” indicates five or more. For each domain, the counts under #Evidence and #Constraints each sum to I. ence evidence is embedded within large volumes of noisy conversational content. Rather than be- ing explicitly stated, user preferences are revealed through accumulated behavioral patterns. For ex- ample, instead of directly stating a preference such as “I prefer concise reports,” a user may repeatedly shorten verbose drafts or prioritize rapid responses under time pressure. Consequently, agents must infer preferences from behavioral evidence from the interaction history. To increase realism, we inject task-irrelevant in- teractions, with task-relevant interactions occurring only once per 100 turns on average, causing infor- mative signals to be deeply buried within noisy his- tories. This signal-to-noise imbalance challenges agents to distinguish meaningful behavioral evi- dence from distractors. Task Instantiation and Validation. We use mini- mal prompts to generate tasks that require agents to execute concrete actions based on the interac- tion history. Task content is newly authored but domain-aligned to prevent trivial reuse. To simu- late real-world agent operations, we include 2,280 API-actionable instances spanning 16 scenarios with parameterized API-call outputs. Table 2 shows the statistics of IBA-BENCH which comprises 6,962 task instances spanning 9 domains across 66 scenarios and 262 preference dimensions, generated using GPT-5.1 at an API cost of approximately $5,000. The distribution is skewed toward non-trivial cases: 1,857 instances involve three or more behavioral evidence items, 5,602 instances contain at least three preference constraints (see Appendix A.1). For quality control, we randomly sample 200 1 https://huggingface.co/BAAI/bge-m3 instances for validation. Across reasonableness, coherence, and persona consistency, human annota- tions achieve 80.2% agreement with a Gwet’sAC 1 of 0.70, while AI-based annotations achieve 94.0% agreement with anAC 1 of 0.93. Based on the vali- dation feedback, we refine the prompts and regen- erate problematic cases, revising about 10% of all instances. Annotation details are in Appendix B.1. 4 Method In this section, we formalize implicit behavioral alignment as a preference-conditioned task exe- cution problem and present IBA-Agent, an LLM- driven personalized agent framework for bridging the gap between historical preference understand- ing and active task execution, as shown in Figure 3. 4.1 LLM-Driven Personalized Agent IBA-Agent takes the current task and the user’s his- torical interactions as input, and follows an iterative evidence-to-action process for personalized task ex- ecution. During this process, an LLM controller maintains the task state, observes the current con- text, reasons over accumulated tool outputs, selects and executes the next tool, and updates the con- text with new evidence or intermediate decisions. Through this loop, the agent identifies dynamic and implicit preference signals from the user’s his- tory and aligns them with the requirements of the current task. The process terminates when the con- troller invokes a task-specific final output head to produce the personalized action. IBA-Agent im- plements this workflow with two complementary modules: DEEP RETRIEVAL and BROAD THINK- ING & DEEP ALIGNMENT, which we detail below. Deep Retrieval. This module uses four retrieval tools to identify task-relevant preference evidence from the large interaction historyH u and dynami- 4 User Longitudinal History •Interaction History •Preference Memory Task Input •Scenario •Instruction •Content Input Context Query Expansion Similarity Retrieval Redundancy Filtering Salience Re-ranking Deep Retrieval Module Observat ion Reason Step 1: Parse and Record Step 2:Decide Next Action Step 3:Choose Toolset Compact Evidence Preference extraction Task-Preference Alignment Broad Thinking & Deep Alignment Module Action Space IBA-Agent Step 4:Execute Action Space Task Content Generation Output Head A Output Head B Here‘s a personalized weekly workout plan for you: cuisine_type: "Sichuan" price_range: "20-30 CNY" dietary_restrictions: "no seafood" Parameter Filling https://meituan.com/waimai/search? cuisine_type=Sichuan& price_range=20-30& flavor_preference= no seafood API Construction Output Tool Selection Tool Execution Check task completion pMon: 30min yoga (low intensity) pTue: 20min brisk walking with stretching; pWed: rest; pThu: 30min Pilates...... Figure 3: Overview of IBA-Agent for personalized task completion. cally build the task-conditioned evidence context C u (x)for downstream reasoning. Given a current task queryx, the agent first retrieves passages us- ing the original task description and then invokes query expansionto generate multiple task-aware sub-queries. As shown in Table 3, these sub- queries probe complementary preference facets. This design allows the same user history to yield task-specific evidence, such as budget trade-offs for purchase tasks and tone, formatting, and revi- sion patterns for writing tasks. Thesimilarity retrievaltool, implemented with BGE-M3, re- trieves semantically relevant passages fromH u . This enables the agent to recover implicit prefer- ence evidence even when the user has not stated a preference in a canonical form. The module also provides evidence refinement tools. Theredundancy filteringtool removes duplicated or highly overlapping retrieved passages using a content-hash heuristic, preventing repeated history snippets from dominating the evidence con- text. Thesalience re-rankingtool uses an LLM-based scorer to prioritize passages with high- confidence preference signals. Each passage is assigned a discrete salience score from 0 to 10, and passages scoring at least 5 are retained for down- stream use; the scoring criteria are summarized in Table 12 in the Appendix. This weighting scheme limits over-interpretation of weak behavioral cues by weighting reliable explicit feedback more heav- ily than noisier implicit behavioral patterns. Broad Thinking & Deep Alignment. This mod- ule uses a set of prompt-guided reasoning tools to transform theC u (x)into a task-specific person- alization plan. Thecompact evidencefunction first organizes the retained passages into a concise QUERY EXPANSION PROMPT You are a search query generator for personalized recommendations. Given a task query, generate diverse retrieval queries to uncover relevant user preferences from historical conversations. Input: Task query information, including the scenario, instruction, and content. Expansion Facets. Facet Retrieval focus q style Tone, wording, expression habits, and preferred writing style. q structure Format, headings, templates, bullet/paragraph preference, length, and detail level. q decision Trade-offs such as cost vs. quality, time vs. thoroughness, or convenience vs. preference. q correction Complaints, corrections, explicit rejections, and disliked suggestions. q habit Repeated choices, default behaviors, and recurring preferences in the scenario. Output: Return target query strings covering multiple facets. No preamble or explanation. Table 3:Prompt-guided query expansion in the Deep Retrieval module. Given a taskx, the gen- erator produces a set of target queriesQ(x) = q style , q structure , q decision , q correction , q habit to re- trieve diverse preference evidence from the history. evidence set by preserving the most informative his- torical interactions. Thepreference extraction function then reads this evidence with respect to the current task and extracts task-relevant prefer- ence signals, such as what the user asked for, cor- rected, rejected, repeatedly chose, or recently up- dated. Thetask-preference alignmentfunc- tion decides how the extracted preferences should guide the current task, producing an alignment plan: response style and structure for generation tasks and constraints or priorities for API-action tasks. 5 ModelWritingWorkDailyPlan.HealthTrans.Med.Leis.Info.Overall • Baseline Models Qwen2.5-7B-Instruct (Qwen et al., 2025)52.751.077.355.567.165.275.040.137.158.3 DeepSeek-V3 (Liu et al., 2024a)65.156.677.162.771.467.378.844.751.964.9 ChatGLM-4-9b-chat (Glm et al., 2024)51.447.370.753.565.458.469.336.737.155.3 QwQ-32B (Qwen Team, 2025)68.761.680.664.575.274.279.647.355.468.4 GPT-4o-mini57.754.977.457.968.168.371.839.942.461.1 GPT-4o (OpenAI, 2024)63.554.175.657.969.566.673.044.143.561.7 GPT-5-mini70.563.976.369.878.370.880.249.863.270.1 GPT-5.1 (OpenAI, 2025)72.962.479.965.673.970.678.056.858.169.2 • Qwen3-4B-Instruct Variants Qwen3-4B-Instruct (Yang et al., 2025)65.260.477.862.873.480.082.952.654.167.6 IBA-Agent (Qwen3-4B-Instruct)80.473.384.077.286.182.389.547.569.278.1 IBA-Agent (Qwen3-4B-Instruct, FC)80.873.983.079.387.781.991.444.969.878.7 • DeepSeek-V3.2 Variants DeepSeek-V3.2 (Liu et al., 2025)68.160.179.863.272.270.779.048.353.666.9 IBA-Agent (DeepSeek-V3.2)83.575.184.576.985.678.087.953.372.378.8 IBA-Agent (DeepSeek-V3.2, FC)86.777.788.979.889.780.292.654.274.181.9 • General-Purpose AI Agents nanobot (DeepSeek-V3.2) (Ren and HKUDS, 2026)69.954.059.362.556.253.059.738.161.158.5 Claude Code (DeepSeek-V3.2) 4 80.564.060.970.363.154.758.133.856.762.9 Hermes Agent(DeepSeek-V3.2) (Nous Research, 2026)81.763.854.270.959.056.959.535.968.663.1 Table 4: Performance comparison using the bge-m3 retriever. Results are evaluated by DeepSeek-V3.2, Qwen3- 4B-Instruct, and Qwen3-30B-A3B-Instruct; scores from Qwen3-4B-Instruct are averaged over five runs. Columns correspond to 9 application domains: Writing (writing and design), Work, Daily (daily consumption), Plan. (planning), Health (exercise and health), Trans. (transportation), Med. (medical services), Leis. (leisure activities), and Info. (information management). Best result is highlighted ingreen bold text, second-best result isunderlined, Function Call (FC) variants are shaded in light purple. (a) Qwen3-4B-Instruct Writing Work Daily Plan. Health Trans. Med. Leis. Info. 50 70 90 (b) DeepSeek-V3.2 Writing Work Daily Plan. Health Trans. Med. Leis. Info. 50 70 90 Qwen3-4B-Instruct IBA-Agent (Qwen3-4B-Instruct) IBA-Agent (Qwen3-4B-Instruct)+ FC DeepSeek-V3.2 IBA-Agent (DeepSeek-V3.2) IBA-Agent (DeepSeek-V3.2)+ FC Figure 4: Radar plots of baseline models and IBA-Agent variants across nine domains. The synthesis step then realizes this plan as the final output, namely personalized content or filled API parameters with an action description. 5 Performance Study All experiments are conducted in an inference-only setting, with no parameter fine-tuning. We evaluate ten modern LLMs spanning the Qwen, DeepSeek, GPT, ChatGLM, and QwQ families, together with three general-purpose agent systems (Claude Code 4 , Hermes Agent (Nous Research, 2026), and nanobot (Ren and HKUDS, 2026)), under a unified RAG pipeline that uses bge-m3 for dense retrieval. Each output is scored against scenario- specific behavioral checkpoints by an LLM judge; for robustness, we aggregate results over three judges (DeepSeek-V3.2, Qwen3-4B-Instruct, and Qwen3-30B-A3B-Instruct). Details of baselines, evaluation metrics, and hyperparameters are pro- vided in Appendix C, with additional experiments on retriever and judge robustness reported in Ap- pendix D. 5.1 Main Results Table 4 reports results on IBA-BENCH. Standard RAG yields moderate and inconsistent results, with gains not strictly monotonic with model scale, sug- gesting that stronger backbones alone are insuffi- cient for personalized task completion. In contrast, IBA-Agent consistently improves over its RAG backbones, increasing DeepSeek-V3.2 from 66.9 to 78.8 and Qwen3-4B-Instruct from 67.6 to 78.1. Adding function calling further boosts performance to 81.9 and 78.7 respectively. This indicates that the key challenge lies not only in retrieving rele- vant memories, but also in converting implicit user preferences into actionable constraints. Compared to the general-purpose agent systems Claude Code, Hermes Agent, and nanobot, IBA- Agent performs substantially better, demonstrat- ing that generic agent workflows transfer poorly to personalized settings. As shown in Figure 4, IBA- Agent achieves broad gains across most task cate- gories, while function calling enables more flexible module invocation and more reliable personalized 4 https://github.com/oboard/claude-code-rev 6 Method VariantsWritingWorkDailyPlan.HealthTrans.MedicalLeisureInfo.Overall IBA-Agent (DeepSeek-V3.2)83.575.184.576.985.678.087.953.372.378.8 w/o Deep Retrieval79.7↓3.867.4↓7.772.9↓11.673.8↓3.179.3↓6.365.1↓12.964.9↓23.046.7↓6.665.9↓6.470.5↓8.3 w/o BT+DA82.7↓0.871.9↓3.283.8↓0.777.4↑0.584.4↓1.274.1↓3.982.7↓5.248.4↓4.970.6↓1.776.3↓2.5 w/o Synth.76.4↓7.167.0↓8.183.1↓1.470.6↓6.381.6↓4.075.8↓2.286.3↓1.648.1↓5.260.6↓11.773.3↓5.5 Table 5: Ablation study of IBA-Agent with DeepSeek-V3.2 as the backbone LLM across task domains, using bge-m3 as the retriever. Results are evaluated by three judges: DeepSeek-V3.2, Qwen3-4B-Instruct, and Qwen3-30B- A3B-Instruct; Qwen3-4B-Instruct scores are averaged over five runs. w/o denotes removing a specific component. BT+DA denotes Broad Thinking + Deep Alignment, and Synth. denotes the original RAG-style synthesis strategy without trajectory-level synthesis. 123 50 60 70 80 ∆=7.3 ∆=5.7 ∆=8.4 Score (%) Standard RAG (QwQ-32B)Standard RAG (GPT-5.1) Standard RAG (GPT-4o)IBA-Agent (w/o Deep Retrieval) IBA-Agent (a) Impact of number of evidence. 12345678 30 40 50 60 70 80 Score (%) DeepSeek-V3.2GPT-5.1 GPT-5-miniIBA-Agent (w/o Broad Thinking) IBA-Agent (b) Impact of number of preferences. StaticDynamic 40 60 80 ∆ = 11.8 ∆ = 12.7 Score (%) Qwen2.5-7B-InstructGPT-4oGPT-5.1 DeepSeek-V3.2QwQ-32BIBA-Agent (c) Impact of preference attributes. QATask 50 60 70 80 90 100 Score (%) DeepSeek-V3DeepSeek-V3.2 QwQ-32B (d) QA vs. task-oriented application. Figure 5: Ablation analyses on evidence quantity, preference complexity, preference attributes, and the knowledge- to-action gap. ∆ denotes the performance gap between IBA-Agent and the second-best model. task completion. 5.2 Ablation Study Table 5 presents the ablation results of IBA-Agent on a 1,000-instance test set obtained through strati- fied sampling from IBA-BENCH. The full model consists of Deep Retrieval, Broad Thinking & Deep Alignment, and trajectory-level synthesis. To as- sess the contribution of each component, we con- sider three variants: removing Deep Retrieval, re- moving Broad Thinking & Deep Alignment, and replacing trajectory-level synthesis with the origi- nal RAG-style synthesis. Among these components, Deep Retrieval has the largest impact: removing it reduces overall per- formance from 78.8 to 70.5 (-8.3), with the largest drops observed in Medical, Transportation, and Daily. This highlights the importance of accu- rately accessing relevant evidence from long in- teraction histories. Removing Broad Thinking & Deep Alignment lowers performance from 78.8 to 76.3 (-2.5). Replacing trajectory-level synthesis with the original RAG-style alternative decreases performance to 73.3 (-5.5), suggesting that effec- tive personalization requires not only retrieving relevant evidence but also integrating it into coher- ent behavioral trajectories. Notably, these results suggest that the modules are effective as a coordi- nated mechanism: without reliable Deep Retrieval evidence, downstream Broad Thinking & Deep Alignment may become counterproductive. 5.3 In-depth Analysis Impact of Number of Evidence. Figure 5a reports preference-conditioned task performance when the number of evidence varies. IBA-Agent consistently outperforms the standard RAG baseline and the variant without Deep Retrieval, highlighting its ability to aggregate preference-relevant signals and the value of broad, deep retrieval. Across all mod- 7 Prompt: IBA-Agent (DeepSeek-V3.2) DeepSeek-V3.2 Domain & Scenario Writing and Design WeChat official account content writing Preference Previously: opening with a viewpoint Now: opening with a story “ I want to write a WeChat public account article about managing work-life balance ......” “Help me write a public account article.” Working from Home: How Can We Protect the Boundary Between Work and Life? Many people think remote work means “freedom”...... I once moved from a stable office job to full-time freelancing,...... Later, I realized that sustaining remote work is not about...... 5 Tactics to Preserve Your Sanity While Working From Home Generate(text) Case 1: Writing (Correct vs Wrong) Domain & Scenario Health Diet plan creation Preference Previously: 3 meals/day dynamic Now: 4 meals/day Prompt: IBA-Agent (DeepSeek-V3.2) DeepSeek-V3.2 “Help me create a diet plan.” Simple weekly diet plan (Three meals per day, no snacks, about 20 minutes of preparation per meal) Generate(text) No further explanation. Here is a basic weekly diet plan for steady energy and focus: •Breakfast(Daily):Oatmeal with nuts...... •Lunch(Daily): A large, colorful salad or ...... •Dinner (Simplify with 15- 20 min prep): Mon/Thu: tofu. Wed/Sun: Whole-wheat pasta....... Monday •Breakfast: Greek yogurt cup: 150g yogurt + 30g oats/granola +...... •Lunch: Quick bean and grain bowl..... •Dinner: Quick tofu and ...... Case 2: DietPlan (Both Wrong) dynamic Remote work offers flexibility, but without clear boundaries, it can lead to extended hour. Here are strategies to manage your work-life balance effectively. •Define Your “Office” Hours:...... •Create Physical & Digital Separation:...... Figure 6: Case studies of personalizedGenerate(text)execution in IBA-BENCH. Each case compares IBA- AGENT (FUNCTION CALL) instantiated with DeepSeek-V3.2 against the same DeepSeek-V3.2 baseline on the same task and with the same dynamic-preference evidence. els, increasing evidence from one to two helps, but expanding to three often hurts, suggesting a trade- off between recovering missing preferences and accumulating noise. Impact of Number of Preferences. Figure 5b shows the performance as the number of prefer- ences (smoothed with a sliding window,w=2) in- creases. Performance drops for all models as con- straints accumulate. IBA-Agent degrades more slowly and widens its lead as preference increases. Removing Broad Thinking & Deep Alignment causes a clear drop, confirming its role in robust multi-constraint alignment. Impact of Preference Attributes. Figure 5c shows the effect of static vs. dynamic preference settings. All models perform better on static preferences, in- dicating easier behavior alignment. For dynamic preferences, IBA-Agent outperforms the strongest baseline by +12.7 points (vs. +11.8 static), demon- strating robustness to preference shifts. QA vs. Task-oriented Application. Figure 5d compares QA with direct task execution. For all models, QA consistently exceeds Task, exposing a clear knowledge-to-action gap: models can state preferences correctly but often fail to enforce them, showing that QA competence does not translate into robust preference-aware task completion. A qualitative illustration of this QA-to-execution dis- tinction is provided in Figure 7 in the Appendix. 5.4 Case Study on Knowledge-to-Action Gap Figure 6 shows twoGenerate(text)cases under dynamic preferences. Both cases use DeepSeek- V3.2 as the shared backbone and compare direct generation against IBA-AGENT on the same task and with the same preference updates. In the writ- ing case, IBA-AGENT correctly applies the user’s latest preference to open with a personal story, whereas the baseline follows the outdated prefer- ence and opens with a viewpoint. This suggests that IBA-AGENT can better isolate dynamic preference cues and synthesize them into actions. However, in the diet-planning case, both methods fail to update the plan from three meals to four meals per day. This indicates that the task remains difficult when the updated preference requires a global structural revision of the output, rather than a localized stylis- tic change. 6 Conclusion In this paper, we identified a critical knowledge-to- action gap in existing personalized agent evalua- tions, which rely heavily on static user profiles and explicit question answering. We argued that true personalization requires agents to synthesize im- plicit signals from longitudinal histories rather than merely retrieving explicit tags. To address this, we introduced IBA-BENCH, a comprehensive bench- mark that evaluates implicit behavioral alignment under dynamic and noisy conditions. We further proposed IBA-Agent, a framework that bridges information retrieval and task execution through deep retrieval and planning-guided generation. Ex- tensive experiments demonstrate that while cur- rent state-of-the-art models struggle to translate historical context into behavioral constraints, our approach significantly improves performance in these complex scenarios. Closing this gap is a pre- requisite for personalized agents in the wild. 8 Limitations We propose IBA-BENCH as a benchmark for eval- uating implicit behavioral alignment in personal- ized agents; however, it still has two main limi- tations. First, IBA-BENCH is constructed using LLM-generated synthetic data. Although the syn- thesis pipeline is carefully designed to inject im- plicit preferences, temporal dynamics, and noise, hallucinated or inconsistent behaviors may still oc- cur. Second, our evaluation focuses on offline, history-conditioned task execution. As a result, the benchmark cannot capture how agents adapt preferences through online interaction or corrective signals. Extending IBA-BENCH to interactive and tool-augmented settings is left for future work. Ethical Considerations Considering potential risks related to privacy and intellectual property, we deliberately avoid using real user logs or information collected from hu- man subjects. All personas, interaction histories, and tasks are fictional, thereby preventing direct exposure of personally identifiable information. At the same time, as LLM-generated content may still reflect biased or undesirable patterns, we address these issues through carefully designed prompts and targeted human review. The benchmark data and prompts will be released under C BY-NC 4.0, and the evaluation scripts will be released under the MIT License. These released materials are in- tended for academic research; any redistribution or derivative use should follow the stated licenses and remain within this research and evaluation scope. References Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 techni- cal report. arXiv preprint arXiv:2303.08774. Saleh Afzoon, Usman Naseem, Amin Beheshti, and Zahra Jamali. 2024. Persobench: Benchmarking personalized response generation in large language models. CoRR, abs/2410.03198. Team Glm, Aohan Zeng, Bin Xu, Bowen Wang, Chen- hui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, and 1 others. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhu- oshu Li, Ziyi Gao, Aixin Liu, and 175 others. 2025. Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. Nat., 645(8081):633–638. Kilem Gwet. 2001. Handbook of inter-rater reliability: How to estimate the level of agreement between two or multiple raters. Gaithersburg, MD: STATAXIS Publishing Company. Yupu Hao, Pengfei Cao, Zhuoran Jin, Huanxuan Liao, Yubo Chen, Kang Liu, and Jun Zhao. 2025. Eval- uating personalized tool-augmented llms from the perspectives of personalization and proactivity. In Proceedings of ACL, pages 21897–21935. Zhiting Hu and Tianmin Shu. 2023. Language mod- els, agent models, and world models: The law for machine reasoning and planning. arXiv preprint arXiv:2312.05230. Hideaki Joko, Shubham Chatterjee, Andrew Ramsay, Arjen P. de Vries, Jeff Dalton, and Faegheh Hasibi. 2024. Doing personal LAPS: llm-augmented di- alogue construction for personalized multi-session conversational search. In Proceedings of ACM SIGIR, pages 796–806. Shawn Li, Chenxiao Yu, Han Wang, Wei Yang, Ryan Rossi, Franck Dernoncourt, Xiyang Hu, Philip Yu, Chaowei Xiao, Huan Zhang, and Yue Zhao. 2026. Fortis: Benchmarking over-privilege in agent skills. Preprint, arXiv:2605.09163. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024a. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingx- uan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, and 1 others. 2025. Deepseek-v3. 2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Ao- han Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, and 3 others. 2024b. Agentbench: Evaluating llms as agents. In The Twelfth ICLR. Mahmoud Mohammadi, Yipeng Li, Jane Lo, and Wendy Yip. 2025. Evaluation and benchmarking of LLM agents: A survey. In Proceedings of ACM SIGKDD, pages 6129–6139. Jisoo Mok, Ik-hwan Kim, Sangkwon Park, and Sun- groh Yoon. 2025. Exploring the potential of llms as personalized assistants: Dataset, evaluation, and anal- ysis. In Proceedings of ACL, pages 10212–10239. 9 Nous Research. 2026.Hermes agent: The agent that grows with you.https://github.com/ NousResearch/hermes-agent. GitHub repository, accessed May 2026. OpenAI. 2024. Hello gpt-4o.https://openai.com/ index/hello-gpt-4o/ . Official announcement of GPT-4o model; accessed: 2026-01-04. OpenAI. 2025. Gpt-5.1: A smarter, more conversational chatgpt.https://openai.com/index/gpt-5-1/. Official announcement of GPT-5.1 model series; ac- cessed: 2026-01-04. Joon Sung Park, Joseph C. O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative agents: Interactive sim- ulacra of human behavior. In Proceedings of UIST, pages 2:1–2:22. Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, and 24 others. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115. Qwen Team. 2025. Qwq-32b: Embracing the power of reinforcement learning.https://qwenlm.github. io/blog/qwq-32b/. Official model introduction blog. Accessed: 2026-01-04. Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. In-context retrieval-augmented lan- guage models. Transactions of the Association for Computational Linguistics, 11:1316–1331. Xubin Ren and HKUDS. 2026. nanobot: The ultra- lightweight personal AI agent.https://github. com/HKUDS/nanobot. GitHub repository, accessed May 2026. Vinay Samuel, Henry Peng Zou, Yue Zhou, Shreyas Chaudhari, Ashwin Kalyan, Tanmay Rajpurohit, Ameet Deshpande, Karthik R. Narasimhan, and Vish- vak Murahari. 2025. Personagym: Evaluating per- sona agents and llms. In Findings of EMNLP, pages 6999–7022. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettle- moyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. In NeurIPS. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learn- ing to summarize with human feedback. Advances in neural information processing systems, 33:3008– 3021. Juntao Tan, Liangwei Yang, Zuxin Liu, Zhiwei Liu, Rithesh R. N., Tulika Manoj Awalgaonkar, Jian- guo Zhang, Weiran Yao, Ming Zhu, Shirley Kokane, Silvio Savarese, Huan Wang, Caiming Xiong, and Shelby Heinecke. 2025. Personabench: Evaluating AI models on understanding personal information through accessing (synthetic) private user data. In Findings of ACL, pages 878–893. Gemini Team. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long con- text, and next generation agentic capabilities. CoRR, abs/2507.06261. Karthik Valmeekam, Matthew Marquez, Sarath Sreed- haran, and Subbarao Kambhampati. 2023. On the planning abilities of large language models - A criti- cal investigation. In NeurIPS. Tiannan Wang, Meiling Tao, Ruoyu Fang, Huilin Wang, Shuai Wang, Yuchen Eleanor Jiang, and Wangchunshu Zhou. 2024. Ai persona: Towards life-long personalization of llms. arXiv preprint arXiv:2412.13103. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompt- ing elicits reasoning in large language models. In NeurIPS. Rebecca Westhäußer, Wolfgang Minker, and Sebastian Zepf. 2025. Enabling personalized long-term interac- tions in llm-based agents through persistent memory and user profiles. arXiv preprint arXiv:2510.07925. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025.Qwen3 technical report.arXiv preprint arXiv:2505.09388. John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering. In NeurIPS. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Ben- gio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answer- ing. In Proceedings of EMNLP, pages 2369–2380. Weizhi Zhang, Xinyang Zhang, Chenwei Zhang, Liang- wei Yang, Jingbo Shang, Zhepei Wei, Henry Peng Zou, Zijie Huang, Zhengyang Wang, Yifan Gao, and 1 others. 2025. Personaagent: When large language model agents meet personalization at test time. arXiv preprint arXiv:2506.06254. Zheng Zhao, Clara Vania, Subhradeep Kayal, Naila Khan, Shay B. Cohen, and Emine Yilmaz. 2025. Per- sonalens: A benchmark for personalization evalua- tion in conversational AI assistants. In Findings of ACL, pages 18023–18055. 10 Jiachen Zhu, Menghui Zhu, Renting Rui, Rong Shan, Congmin Zheng, Bo Chen, Yunjia Xi, Jianghao Lin, Weiwen Liu, Ruiming Tang, and 1 others. 2025. Evo- lutionary perspectives on the evaluation of llm-based ai agents: A comprehensive survey. arXiv preprint arXiv:2506.11102. Appendix A Benchmark Details A.1 Dataset Details Table 13 summarizes the statistics across domains and scenarios, including the number of preference dimensions (D), preference values (P), task in- stances (I), dynamic-preference support, and ac- tion space. This provides a quantitative view of IBA-BENCH’s richness in cross-domain scenario coverage and preference-space diversity. Figure 8 shows the domain and scenario distributions. Table 14 provides illustrative examples of how preference dimensions are defined in our bench- mark across the Writing and Design domain. It is organized in a domain–scenario–dimension–value hierarchy, and lists mutually exclusive preference values for each dimension to support task instantia- tion and controlled evaluation. A.2 Task Formulation and Evaluation Details This section provides the detailed evaluation pro- tocol for the three final-action types inY. For all types, the evaluator focuses on the final ac- tion rather than whether the agent can merely de- scribe or recall user preferences. Table 13 re- ports the action type of each scenario, including 50 Generate(text)and 16CallAPI(params)sce- narios. Generate(text) .This type is used for text- oriented tasks where the agent completes the re- quest by producing a free-form personalized re- sponse, such as a plan, explanation, or written mes- sage. Each instance contains a set of hidden per- sonalized requirements for a specific user, such as style preferences, structural preferences, or situa- tional needs. The evaluator assigns a binary score to each requirement according to whether the gen- erated response satisfies it, and the instance score is the average over all requirements. An example is shown in Table 15. CallAPI(params). This type evaluates whether an agent can translate preferences into executable tool parameters. In scenarios such as ordering food or booking appointments, the agent’s final action is represented as structured API parameters that can be passed to the corresponding tool interface to per- form the intended operation. Each instance defines an API template with controllable slots, and the hid- den personalized requirements specify which slots should reflect the user’s preferences and current sit- uation. The evaluator scores these personalization- relevant slots independently: each slot receives a binary score indicating whether the submitted value satisfies the user’s preferences in the given scenario, and the instance score is the average over all scored slots. Task-fixed or API-validity-only parameters are excluded from the personalization score. An example is shown in Table 16. A.3 Prompt Details Figure 9 shows the prompt template used to gen- erate casual-chat dialogues in IBA-BENCH. Fig- ures 10 and 11 show the prompt templates used to synthesize user historical dialogue evidence. BBenchmark Annotation and Validation B.1 Human Annotation Protocol To evaluate the quality of the generated benchmark instances, we randomly sampled 200 instances for manual validation. Each instance was indepen- dently annotated by two human annotators along three binary dimensions: reasonableness, coher- ence, and persona consistency. These dimensions are designed to assess whether an instance is rea- sonable, internally consistent, and aligned with the intended user role and evolving preferences over time. Reasonableness.Whether the interaction history and instantiated task are reasonable under common sense and the task setting, without obvious hallu- cinations or unrealistic assumptions. 1: plausible and credible. 0: clearly implausible or severely hallucinatory. Coherence. Whether the interaction history and task are logically connected and internally self- consistent, without major contradictions or abrupt discontinuities. 1: coherent and understandable. 0: conflicting, broken, or inconsistent. Persona Consistency. Whether the instance is consistent with the target role and appropriately reflects preference changes over time. 1: consistent with the intended role and preference evolution. 0: inconsistent with the role or misrepresents prefer- ence dynamics. Annotation procedure.The two human annota- tors independently labeled all sampled instances 11 ModelWritingWorkDailyPlan.HealthTrans.Med.Leis.Info.Overall • Baseline Models Qwen2.5-7B-Instruct (Qwen et al., 2025)49.349.577.056.768.168.372.940.233.757.8 DeepSeek-V3 (Liu et al., 2024a)64.157.277.767.074.271.779.350.251.266.5 ChatGLM-4-9b-chat (Glm et al., 2024)45.645.270.954.366.260.669.038.033.054.4 QwQ-32B (Qwen Team, 2025)67.262.281.368.977.979.080.353.456.070.2 GPT-4o-mini54.754.178.261.369.071.571.745.239.261.6 GPT-4o (OpenAI, 2024)58.853.076.761.370.570.673.149.139.862.0 GPT-5-mini73.266.076.475.781.974.181.952.468.073.1 GPT-5.1 (OpenAI, 2025)75.863.280.270.275.574.078.858.059.171.1 • DeepSeek-V3.2 Variants DeepSeek-V3.2 (Liu et al., 2025)69.461.380.367.673.775.479.654.155.869.1 IBA-Agent (DeepSeek-V3.2)86.378.385.382.186.580.787.360.379.981.9 IBA-Agent (DeepSeek-V3.2, FC)88.379.689.282.990.484.092.259.878.784.0 • Qwen3-4B-Instruct Variants Qwen3-4B-Instruct (Yang et al., 2025)63.659.580.166.174.776.081.548.953.967.8 IBA-Agent (Qwen3-4B-Instruct)82.475.184.481.787.5 82.588.849.973.680.0 IBA-Agent (Qwen3-4B-Instruct, FC)81.176.083.183.588.582.590.050.072.080.2 • General-Purpose AI Agents nanobot (DeepSeek-V3.2)69.752.760.467.461.159.460.444.867.161.2 Claude Code (DeepSeek-V3.2)75.159.560.369.562.359.157.138.757.361.8 Hermes Agent (DeepSeek-V3.2)79.160.261.371.762.859.360.341.769.664.5 Table 6: Performance comparison using the bge-m3 retriever and Qwen3-30B-A3B-Instruct as the judge model. The columns correspond to nine application domains: Writing (writing and design), Work, Daily (daily con- sumption), Plan. (planning), Health (exercise and health), Trans. (transportation), Med. (medical services), Leis. (leisure activities), and Info. (information management). The best result in each column is highlighted asgreen bold text, the second-best result isunderlined,Function Call variantsare shaded in light purple. FC stands for Function Call. ModelWritingWorkDailyPlan.HealthTrans.Med.Leis.Info.Overall • Baseline Models Qwen2.5-7B-Instruct (Qwen et al., 2025)42.742.075.045.857.960.773.932.832.451.0 DeepSeek-V3 (Liu et al., 2024a)48.243.274.750.361.262.275.634.640.155.0 ChatGLM-4-9b-chat (Glm et al., 2024)42.139.067.243.856.554.767.129.831.848.5 QwQ-32B (Qwen Team, 2025)54.550.077.254.167.169.376.339.343.159.6 GPT-4o-mini47.144.274.247.159.764.169.430.734.253.2 GPT-4o (OpenAI, 2024)55.142.072.045.260.862.369.935.933.953.3 GPT-5-mini57.352.473.458.671.166.176.642.548.161.3 GPT-5.1 (OpenAI, 2025)58.449.076.254.266.565.775.849.043.459.8 • DeepSeek-V3.2 Variants DeepSeek-V3.2 (Liu et al., 2025)54.647.276.952.564.464.777.039.340.558.0 IBA-Agent (DeepSeek-V3.2)72.264.883.569.382.573.487.142.557.071.8 IBA-Agent (DeepSeek-V3.2, FC)78.469.387.875.388.176.392.443.962.676.7 • Qwen3-4B-Instruct Variants Qwen3-4B-Instruct (Yang et al., 2025)52.349.471.051.965.588.082.959.543.760.6 IBA-Agent (Qwen3-4B-Instruct)71.965.282.471.082.9 80.289.639.658.072.6 IBA-Agent (Qwen3-4B-Instruct, FC)72.665.180.672.484.179.991.136.360.673.0 • General-Purpose AI Agents nanobot (DeepSeek-V3.2)58.444.854.351.945.645.556.928.445.849.3 Claude Code (DeepSeek-V3.2)81.565.456.967.658.450.556.426.856.560.9 Hermes Agent (DeepSeek-V3.2)82.163.435.267.848.654.655.528.267.157.5 Table 7: Performance comparison using the bge-m3 retriever and DeepSeek-V3.2 as the judge model. The columns correspond to nine application domains: Writing (writing and design), Work, Daily (daily consumption), Plan. (planning), Health (exercise and health), Trans. (transportation), Med. (medical services), Leis. (leisure activities), and Info. (information management). The best result in each column is highlighted asgreen bold text, the second-best result isunderlined,Function Call variants are shaded in light purple. FC stands for Function Call. 12 ModelWritingWorkDailyPlan.HealthTrans.Med.Leis.Info.Overall • Baseline Models Qwen2.5-7B-Instruct (Qwen et al., 2025)66.061.579.864.075.266.578.247.345.366.0 DeepSeek-V3 (Liu et al., 2024a)83.169.378.870.878.868.181.449.264.473.1 ChatGLM-4-9b-chat (Glm et al., 2024)66.657.674.062.373.460.071.742.346.563.1 QwQ-32B (Qwen Team, 2025)84.472.683.470.480.674.482.149.367.075.3 GPT-4o-mini71.366.479.965.275.669.274.343.753.768.4 GPT-4o (OpenAI, 2024)76.567.478.067.277.367.075.947.456.769.7 GPT-5-mini81.0273.3479.2075.1682.1272.3282.2654.7873.5876.10 GPT-5.1 (OpenAI, 2025)84.675.083.272.579.772.279.563.371.976.7 • DeepSeek-V3.2 Variants DeepSeek-V3.2 (Liu et al., 2025)80.371.882.369.578.571.980.351.664.673.7 IBA-Agent (DeepSeek-V3.2)92.182.384.879.487.779.989.257.180.082.8 IBA-Agent (DeepSeek-V3.2, FC)93.384.189.781.390.680.493.359.081.185.1 • Qwen3-4B-Instruct Variants Qwen3-4B-Instruct (Yang et al., 2025)79.772.382.470.580.176.184.449.564.974.6 IBA-Agent (Qwen3-4B-Instruct)86.879.685.279.088.084.190.153.076.181.6 IBA-Agent (Qwen3-4B-Instruct, FC)88.880.785.382.290.583.493.248.476.982.9 • General-Purpose AI Agents nanobot (DeepSeek-V3.2)81.564.663.268.362.054.261.841.070.564.9 Claude Code (DeepSeek-V3.2)84.867.265.673.868.654.560.735.956.266.1 Hermes Agent (DeepSeek-V3.2)83.867.866.173.365.656.762.837.869.067.4 Table 8: Performance comparison using the bge-m3 retriever and Qwen3-4B-Instruct as the judge model. Reported results are averaged over five runs. The columns correspond to nine application domains: Writing (writing and design), Work, Daily (daily consumption), Plan. (planning), Health (exercise and health), Trans. (transportation), Med. (medical services), Leis. (leisure activities), and Info. (information management). The best result in each column is highlighted asgreen bold text, the second-best result isunderlined,Function Call variantsare shaded in light purple. FC stands for Function Call. using the criteria described above. In addition, we obtained two independent AI-based annotations from GPT-5.4 using the same evaluation dimen- sions and instructions. To assess the reliability of the validation process, we compared (1) the agree- ment between the two human annotators and (2) the agreement between the two AI-based annotations. Human annotations were treated as the primary ref- erence, while AI judgments served as an additional external comparison signal. Table 11 reports two complementary measures of annotation reliability: raw agreement and Gwet’s AC 1 (Gwet, 2001). Raw agreement is the propor- tion of instances receiving the same label from two annotators: P o = 1 N N X i=1 1(y (1) i = y (2) i ), whereNis the number of annotated instances,y (1) i andy (2) i denote the labels assigned by two raters to instancei, and1(·)is the indicator function. Be- cause the labels are imbalanced, we additionally report Gwet’sAC 1 , which is more robust than Co- hen’s kappa under skewed label distributions: AC 1 = P o − P e 1− P e whereP o is the observed agreement andP e is the chance agreement estimated under Gwet’s formu- lation. C Experiment Settings C.1 Baselines We evaluate personalized task completion across a diverse set of LLMs, including Qwen2.5-7B- Instruct (Qwen et al., 2025), ChatGLM-4-9B- Chat (Glm et al., 2024), DeepSeek-V3 (Liu et al., 2024a), DeepSeek-V3.2 (Liu et al., 2025), QwQ- 32B (Qwen Team, 2025), GPT-4o-mini, GPT- 4o (OpenAI, 2024), GPT-5-mini, GPT-5.1 (Ope- nAI, 2025), and Qwen3-4B-Instruct (Yang et al., 2025), under a retrieval-augmented generation (RAG) baseline. All models are accessed via offi- cial or widely used inference platforms (DeepSeek API, SiliconFlow, and OpenAI API). The RAG baseline uses bge-m3 1 to retrieve sentence-level chunks concatenated with the task query as LLM input. Beyond standalone LLMs, we also evaluate representative agent-based systems, namely Claude 13 ModelWritingWorkDailyPlan.HealthTrans.Med.Leis.Info.Overall Qwen2.5-7B-Instruct66.0±0.5761.5±0.3079.8±0.4764.0±0.2975.2±0.1566.5±0.3478.2±0.1447.3±0.6245.3±0.5366.0±0.22 DeepSeek-V383.1±9.4869.3±0.4278.8±0.2370.8±0.2878.8±0.3868.1±0.1781.4±0.2149.2±0.4264.4±0.1173.1±1.28 ChatGLM-4-9b-chat66.6±0.6757.6±0.3774.0±0.5562.3±0.1873.4±0.4860.0±0.1671.7±0.3142.3±0.8346.5±0.8363.1±0.19 QwQ-32B84.4±8.7472.6±0.4083.4±0.4570.4±0.1880.6±0.1674.4±0.3682.1±0.1549.3±0.2567.0±0.1975.3±1.06 GPT-4o-mini71.3±0.6566.4±0.3079.9±0.4365.2±0.3075.6±0.9769.2±0.2474.3±0.1843.7±1.0853.7±0.4868.4±0.29 GPT-4o76.5±0.6167.4±0.3078.0±0.3267.2±0.1877.3±0.5267.0±0.0475.9±0.2647.4±0.2356.7±0.2369.7±0.09 GPT-5-mini81.02±9.7673.34±7.7679.20±0.6675.16±5.0782.12±4.1572.32±2.0982.26±0.5054.78±3.3473.58±11.0176.10±4.64 GPT-5.184.6±0.6175.0±0.0783.2±0.3072.5±0.5479.7±0.1572.2±0.3579.5±0.0763.3±0.6771.9±0.9076.7±0.11 DeepSeek-V3.2 Variants DeepSeek-V3.280.3±0.6871.8±0.3382.3±0.5669.5±0.1978.5±0.6171.9±0.2680.3±0.3551.6±1.0464.6±0.5273.7±0.24 IBA-Agent (DeepSeek-V3.2)92.1±0.8882.3±0.34 84.8±3.2079.4±0.1387.7±0.2879.9±5.4889.2±0.7057.1±6.4780.0±0.6382.8±0.31 IBA-Agent (DeepSeek-V3.2, FC) 93.3±0.5684.1±0.3189.7±0.3881.3±0.3890.6±0.2380.4±0.3893.3±0.1359.0±0.3381.1±0.9685.1±0.23 Qwen3-4B-Instruct Variants Qwen3-4B-Instruct79.7±0.3872.3±0.3682.4±0.4170.5±0.2280.1±0.2876.1±0.5584.4±0.8149.5±0.6164.9±0.47 74.6±0.15 IBA-Agent (Qwen3-4B-Instruct)86.8±0.3079.6±0.4385.2±4.7579.0±0.2788.0±0.1384.1±3.1490.1±0.1853.0±7.7676.1±0.5781.6±0.24 IBA-Agent (Qwen3-4B-Instruct, FC) 88.82±0.7880.74±0.3585.34±0.4382.22±0.3090.54±0.2683.46±0.2393.20±0.1648.46±0.7876.96±0.69 82.90±0.29 General-Purpose AI Agents nanobot81.5±0.464.6±0.763.2±1.168.3±0.362.0±0.254.2±0.561.8±0.241.0±0.670.5±1.1 64.9±0.2 Hermes Agent83.8±0.367.8±0.566.1±0.273.3±0.465.6±0.556.7±0.262.8±0.637.8±0.769.0±0.367.4±0.1 Claude Code84.8±4.767.2±0.565.6±0.673.8±0.368.6±0.354.5±0.460.7±0.235.9±0.656.2±0.566.1±0.5 Table 9: Performance comparison using the bge-m3 retriever and Qwen3-4B-Instruct as the judge model. Reported results are averaged over five runs, with sample standard deviations shown in a compact±format. The columns correspond to nine application domains: Writing (writing and design), Work, Daily (daily consumption), Plan. (planning), Health (exercise and health), Trans. (transportation), Med. (medical services), Leis. (leisure activities), and Info. (information management).Greenhighlights the Overall column,yellowmarks cells with moderate variation (std≥ 2.0and< 5.0), andredmarks cells with high variation (std≥ 5.0). FC stands for Function Call. ModelWritingWorkDailyPlan.HealthTrans.MedicalLeisureInfo.Overall Qwen2.5-7B-Instruct46.945.676.653.266.685.779.351.337.159.2 DeepSeek-V354.248.375.966.172.589.787.353.443.164.7 GPT-4o-mini49.945.979.054.970.092.890.558.837.662.4 GPT-4o49.845.477.454.366.691.887.852.035.860.8 Table 10: Performance comparison using all-mpnet-base-v2 retriever. The columns correspond to nine application domains: Writing (writing and design), Work, Daily (daily consumption), Plan. (planning), Health (exercise and health), Trans. (transportation), Med. (medical services), Leis. (leisure activities), and Info. (information management). DimensionAgreementAC 1 Human–Human Reasonableness73.2%0.61 Coherence82.5%0.79 Persona Consistency85.1%0.72 Overall80.2%0.70 AI–AI (GPT-5.4) Reasonableness94.5%0.94 Coherence94.5%0.94 Persona Consistency93.0%0.92 Overall94.00%0.93 Table 11: Human–human and AI–AI agreement on benchmark quality annotations over 200 matched sam- ples. Code, Hermes Agent, and nanobot (a lightweight personal AI agent inspired by OpenClaw), to as- sess their end-to-end task-completion performance under the same personalized setting. Signal categoryScore contribution Explicit revisions3–4 User complaint or strong affect3–4 Direct preference statement2 Implicit behavioral patterns1 Retention threshold≥5 Table 12: Scoring criteria used by thesalience re-rankingtool for preference-aware passage re- ranking. Implicit behavioral patterns are indirect cues, such as repeated behaviors. C.2 Evaluation Metric For each scenario, we define a small set of be- havioral checkpoints, each corresponding to an atomic seed preference of the user in that sce- nario. For generation-based tasks, given an output y i for instancei, we use three LLM-based judges, namely DeepSeek-V3.2, Qwen3-4B-Instruct, and Qwen3-30B-A3B-Instruct, to assess whethery i satisfies each checkpoint, yielding a binary score 14 What do l preferbetween the four-part structure (content→ viewpoint →example →reflection) and the concise bullet-point style for my reading notes? concise bullet-point style “prompt": instruction": "Help me organize my reading notes. Here is the basic situation:" “content": “Notes on \"Deep Work\" by Cal Newport examine the benefits of focused, distraction-free work in a world full of digital noise. Key concepts include the importance of cultivating deep concentration......”. , Reading Notes: Deep Work by Cal Newport (Personalized for Amina) Core Takeaway: Deep work is the ability to focus...... Key Concepts & My Reflections: 1.The Deep Work Hypothesis •Concept: Concentration is ..... •My Reflection: Directly applies to ...... QA Input QA Output Task InputTask Output Figure 7: QA (using HiCUPID (Mok et al., 2025) as an example) vs. IBA-Agent for preference modeling. All comparisons are conducted using DeepSeek-V3.2 to control for model capability. IBA-Bench Figure 8: Hierarchical distribution of domains and sce- narios in the benchmark. f k i,j (y i )∈0, 1. The instance-level score is com- puted as the average satisfaction across checkpoints. For API-actionable tasks, each required param- eter is evaluated as a checkpoint, receiving1if it is valid and consistent with the task require- ment and inferred user preference, and0otherwise. The instance-level score is the average correctness across required parameters. C.3 Parameters We evaluate all models using three judges: DeepSeek-V3.2, Qwen3-4B-Instruct, and Qwen3- 30B-A3B-Instruct, with the maximum output length set to 3,000 tokens. For Qwen3-4B-Instruct, we report the average over five runs to assess eval- uation stability, with results shown in Table 9 in Appendix A. For retrieval, we use bge-m3 with 256-token chunks and a 50-token overlap. Stan- dard retrieval returns the top-3 passages per query, while Broad Retrieval issues up to five expanded queries and retains at most ten passages after fil- tering and optional re-ranking. We fix the random seed to 42 for reproducibility. D Supplementary Experiments Table 10 reports supplementary benchmark results using theall-mpnet-base-v2retriever, evaluat- ing Qwen2.5-7B-Instruct, DeepSeek-V3, GPT-4o- mini, and GPT-4o. Compared with thebge-m3set- ting in Table 4,all-mpnet-base-v2improves the overall score for all four models (+6.1 for Qwen2.5- 7B, +7.6 for DeepSeek-V3, +4.4 for GPT-4o-mini, and +3.1 for GPT-4o), with the largest gains con- centrated in the Planning and Health domains. Ta- bles 6, 7, and 8 present evaluation results on the same set of execution outputs using three different judge models, namely Qwen3-30B-A3B-Instruct, DeepSeek-V3.2, and Qwen3-4B-Instruct. Among them, Table 9 further reports the per-cell standard deviation across five repeated evaluations under Qwen3-4B-Instruct as the judge. The results show that overall scores remain highly stable for most models and IBA-Agent variants, with overall stan- dard deviations generally below 0.31, suggesting that the main performance trends are highly repro- ducible. Across three judge models, IBA-Agent consistently outperforms all baseline models, and the function-calling variant further improves over- all performance in most settings, demonstrating the robustness of structured history modeling and modular agent execution. 15 DomainScenarioDPIDyn.Action Space Writing and Design Peer review reports4954✓Generate(text) Academic paper writing61571✓ Generate(text) Research proposal (RP) writing41376✓Generate(text) Business plan writing3829✓ Generate(text) Diary writing3723✓Generate(text) Technical documentation writing41280✓Generate(text) Blog post writing244✗ Generate(text) WeChat public account article3827✓Generate(text) Novel writing246✗Generate(text) Resume writing51472✓ Generate(text) Short video script design31141✗Generate(text) PPT slide content writing2915✓ Generate(text) Mind map design42079✓Generate(text) Modular course writing (teaching aid)41471✓ Generate(text) Instruction manual writing31573✓ Generate(text) Self-introduction script writing3829✗Generate(text) Work Scholarship application717140✓Generate(text) Performance report spreadsheet62390✓Generate(text) Schedule planning631119✓ Generate(text) Data analysis427101✓Generate(text) Programming52390✓ Generate(text) Leave application41093✗Generate(text) Project management522105✓Generate(text) Communication management257✗ Generate(text) Announcement drafting31274✓Generate(text) Event organization414111✗Generate(text) Work summary writing416113✓Generate(text) Terminology explanation314112✓Generate(text) Daily Consumption Food ordering315122✓CallAPI(params) Ticket booking417145✓CallAPI(params) Clothing purchase415153✓ CallAPI(params) Digital product purchase41089✗CallAPI(params) Accommodation booking412138✗CallAPI(params) Online course purchase512154✗CallAPI(params) Book purchase412127✓ CallAPI(params) Renting a house1128158✗CallAPI(params) Planning Weekly plan419122✓ Generate(text) Monthly plan412107✓Generate(text) Long-term goal setting316130✓Generate(text) Project plan415114✗Generate(text) Financial plan31374✓Generate(text) Travel plan518123✓ Generate(text) Study plan419154✓Generate(text) Career development plan416100✓Generate(text) Habit-building plan418115✗Generate(text) Skill improvement plan416112✓ Generate(text) Exercise and Health Exercise plan522260✓Generate(text) Diet plan421255✓ Generate(text) Sleep plan632262✓Generate(text) Mental health support164✗Generate(text) Transportation Commute route planning521236✗ CallAPI(params) Taxi / ride-hailing travel268✗ CallAPI(params) Travel and sightseeing route planning419215✗CallAPI(params) Parking31059✗CallAPI(params) Medical Services Appointment registration415210✗CallAPI(params) Medicine selection417231✗CallAPI(params) Online medical consultation31483✗Generate(text) Leisure Activities Audio-visual entertainment424229✓CallAPI(params) Outdoor leisure256✗CallAPI(params) Social assistance31498✓Generate(text) Information Management Research report writing418174✓Generate(text) Timeline organization31041✓Generate(text) Meeting minutes organization31360✓Generate(text) Reading notes2916✓Generate(text) Folder organization519191✗Generate(text) Tool usage934182✓ Generate(text) Table 13: Scenario-level statistics across benchmark domains. D: number of preference dimensions; P: number of preference values defined for the scenario; I: number of task instances; Dyn.: whether dynamic preferences are supported. 16 DomainScenarioPreference DimensionPreference Value Writing and DesignJournal writing Time granularity Records by day (one entry per whole day). Records by key moments/timestamps. Content structure Prefers a fixed template: “What happened / How I felt”. Prefers free-form writing (write whatever comes to mind). Preferred the fixed template; now prefers free-form writing. Length preference Prefers short entries (about 50–150 words each). Prefers long entries (detailed expansion; 300+ words). Writing and DesignBusiness plan writing Overall narrative style Prefers narrative-driven writing. Prefers logic-driven writing (clear structure, data-centric). Preferred logic-driven writing; now narrative-driven writing. Preferred narrative-driven writing; now logic-driven writing. Market analysis depth Prefers extensive use of industry data. Prefers highlighting key data points with concise interpretation. Risk section writing Emphasizes explicit identification of risks. Follows an optimistic narrative, downplays risks. Writing and DesignTechnical doc writing Document structure Task-oriented structure organized around “how-to” steps. Concept-oriented structure. Technical depth Prefers detailed explanations. Prefers simplified explanations. Example code style Prefers complete, runnable code examples. Prefers code snippets (key parts only). Prefers pseudocode. Previously preferred complete; now prefers pseudocode. Previously preferred pseudocode; now prefers code snippets. Previously preferred code snippets; now prefers pseudocode. API explanation Prefers table-style specification. Prefers narrative explanation of API behavior. Writing and DesignReview writing Overall structure Prefers organizing reviews into Summary / Pros / Cons sections. Prefers unstructured summaries with weaknesses separated. Weakness format Prefers bullet-pointed weaknesses (with numbering). Prefers bullet-pointed weaknesses (without numbering). Weakness count Requires exactly three weaknesses. Requires exactly five weaknesses. Weakness content Prefers keywords + full sentences. Prefers direct full-sentence descriptions. Preferred direct sentences; now prefers keywords + sentences. Writing and DesignBlog article writing Narrative perspective First-person personal perspective. Third-person objective perspective. Citations / sources Prefers academic style: formal citations, links, references. Prefers original style. Writing and DesignPublic account writing Opening style Storytelling-based opening. Viewpoint-based opening. Preferred storytelling-based opening, now viewpoint-based. Preferred viewpoint-based opening, now storytelling-based. Social expression Leans toward social topics. Leans toward personal perspectives. Title style Attention-grabbing titles. Formal titles (standard phrasing, fewer stylistic markers). Writing and DesignNovel writing Narrative perspective First-person narration. Third-person narration. Narrative type Linear storytelling in chronological order. Non-linear storytelling (e.g., flashbacks, parallel storylines). Table 14: Examples of preference dimensions and values in the Writing and Design domain. 17 EXAMPLE 1: Generate(text) Output typeScenarioCurrent query Generate(text)Diet planningHelp me create a diet plan. Hidden evaluation instance. The generated plan should satisfy three personalized requirements: (1) muscle gain with about+300kcal/day; (2) low fat, with fat intake≤ 20% of total calories; (3) breakfast is mandatory. Agent output (summary version). I will make breakfast a central, non-negotiable part of the plan so it supports your muscle gain and keeps your day structured. Eat about+300kcal above maintenance overall, with fat kept at or below20%of total calories. Use a fixed daily breakfast within 60–90 minutes of waking, targeting about 400–550 kcal and 25–30 g protein. Breakfast options include protein oatmeal, an egg-and-toast plate, or a low-fat smoothie. Lunch and dinner use lean protein, rice/quinoa/potatoes, vegetables, and measured oil; dinner may include a small portion of avocado or nuts. Behavioral score. CriterionScoreEvaluator rationale Muscle gain with +300 kcal/day1The plan provides a +300 kcal surplus and supports muscle gain. Low fat,≤ 20% of calories0Fat sources such as seeds, avocado, nuts, and oil may exceed the fat cap. Breakfast is mandatory1Breakfast is explicitly included as a detailed daily meal. Total / Average2/3Average score: 0.67 Table 15: Example of a Generate(text) task in IBA-BENCH. EXAMPLE 2: CallAPI(params) Output typeScenarioCurrent query CallAPI(params)Food deliveryHelp me order a food delivery. Hidden evaluation instance. The API action should satisfy three personalized requirements: (1) the user used to like spicy delivery food, but has recently caught a cold and cannot eat spicy food; (2) the user prefers Western cuisine; (3) the preferred price range is 20–30 yuan. Agent output (summary version). Filled API template https://meituan.com/waimai/search? cuisine_type=Western& meal_type=lunch& price_range=under 30& flavor_preference=mild& dietary_restrictions=non-spicy, no seafood& restaurant_name=none& quantity=1& time_specific_preference=none Reasoning: The agent overrides the user’s previous spicy-food preference because the user recently caught a cold. It therefore sets mild and non-spicy. It also preserves the user’s Western cuisine preference and budget by setting under 30. Long-term preference / budget Dynamic health-related update Other parameters Behavioral score. CriterionScoreEvaluator rationale Avoid spicy food due to recent cold1Uses mild and non-spicy. Prefer Western cuisine1Sets cuisine_type=Western. Price range 20–30 yuan1Sets price_range=under 30. Total / Average3/3Average score: 1.00 Table 16: Example of a CallAPI(params) task in IBA-BENCH. 18 SYSTEM PROMPT FOR CASUAL-CHAT GENERATION Role Setting. You are an English casual-chat dialogue generation assistant. The current user’s name is “user_name,” and their general mood/state is “mood.” Their detailed persona is as follows: persona_summary Task. Your task is to generate natural, realistic daily casual-chat content based solely on the personal information in the above user persona (such as age, major, school, living environment, interests, emotions, etc.). Important requirements: 1. The conversation should only be about light everyday topics, such as study rhythm, life status, social life, hobbies, emotional feelings, city/campus experiences, etc. 2. Absolutely no specific tasks or instructions, especially anything like “help me review a paper,” “write reviewer comments,” “this paper/manuscript/research,” “task/instruction/system prompt,” etc. 3. Even if the user persona includes research direction, paper topics, or project background, these can only serve as background atmosphere — do not expand on research/paper content or switch into a “task-helping” mode. 4. The conversation may include only two speaker prefixes: “User:” and “ChatGPT:”. No other notes or markers. 5. The overall tone should be natural and unforced — more like casual chatting between friends than executing a task. Figure 9: System prompt used for casual-chat generation. USER PROMPT FOR EVIDENCE DATA GENERATION INPUTS User Information. person_info User Preferences (Hidden — for the scriptwriter only). instance Task Overview. instruction_text Domain / Category. domain Sub-scenario. sub_scenario Task Context / Available Materials. content_text TASK Generate a complete, natural, and fluent English conversation in which a user and “ChatGPT” collaboratively accomplish: scenario Generate the full dialogue now. Figure 10: User prompt used for evidence data generation. 19 SYSTEM PROMPT FOR EVIDENCE DATA GENERATION You are a dialogue-generation assistant. Generate a complete, natural, and fluent English conversation between a user and “ChatGPT” that collaboratively accomplishes the user’s task. Core Premise (Non-negotiable). • Only the generator (you) can see hidden preferences. • The ChatGPT character in the dialogue cannot see hidden preferences. • ChatGPT may revise outputs only based on: – what the user says in the dialogue, and – what ChatGPT produced in the previous turn. Dialogue Format (Strict). • Use only two speaker labels: User: and ChatGPT:. • The dialogue is entirely in English. • Exactly turn_count turns in total, alternating lines starting with User: then ChatGPT:. • Each turn contains one speaker line only; no narration, no stage directions, no extra labels, no blank lines. Hard Rules. 1. First user turn (no preference disclosure). The first User: turn states only the task objective and necessary context/materials. It must not mention hidden preferences, “instance”, or any style or format constraints. 2. Immediate draft after turn 1. After the first user message, ChatGPT must produce an initial complete deliverable. No clarification questions about preferences, formatting, structure, tone, or style are allowed. 3. Multi-round revision is mandatory. After Draft 1, the user provides revisions over several turns. Each ChatGPT response must update the content rather than repeat prior versions. 4. Preference points per user turn. Starting only after Draft 1, each new user turn introduces one to three preference points. Preferences must be expressed implicitly and never stated explicitly. 5. User voice constraints. The user uses direct commands, not polite questions. Forbidden expressions include “Could you. . . ”, “Can you. . . ”, “Would you mind. . . ”, and “Is it possible. . . ”. User feedback should be brief (one to two sentences). 6. No meta-leaks. ChatGPT must never mention hidden preferences, “instance”, or prompt mechanics. 7. Coverage requirement. By the final turn, the output must satisfy all hidden preference points. Every distinct preference point must appear explicitly and concretely somewhere in the dialogue. 8. Materials handling (conditional). If required source materials are provided, include them verbatim and in full. 9. Deliverable type. deliverable_type_instruction 10. No raw placeholders. Do not output placeholders such as “[Company Name]”. If specifics are missing, omit them or use a plausible generic name. 11. Response length. Each ChatGPT: turn should be concise, typically under 230 words. Dialogue Flow (Required). • Task entry: user starts directly with the objective. • Draft 1: ChatGPT produces a full first draft immediately. • Iterative refinement: user reveals preferences gradually; ChatGPT revises accordingly. • End with a short user closing line such as “Alright, this works.” Output Requirement. Generate the complete dialogue in one go, following all constraints above. Hidden Preference Instance. instance Figure 11: System prompt used for evidence data generation. 20