Paper deep dive
MIMESIS: Learning User Simulators as Training Environments for Interactive Agents
Hoang Phan, Dat Huynh, Andrey Zhmoginov, Qi Zeng, Wancen Mu, Yue Cao, Shengjie Bi, Yun He, Changdae Oh, Deren Lei
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/11/2026, 2:31:56 AM
Summary
The paper introduces MIMESIS, a purpose-built user simulator (4B and 9B parameters) designed to train and evaluate interactive language agents. Unlike off-the-shelf LLMs that are overly cooperative, MIMESIS is trained on human conversations with explicit reasoning supervision (ThoughtTrace) and 13 realistic behavioral patterns. It achieves superior performance on benchmarks like SOUL-Index and RealUserSim. The paper also proposes Coached On-Policy Self-Distillation (CSD), a method that uses the simulator's private reasoning and future reactions to provide dense, token-level coaching feedback to the agent during reinforcement learning, improving agent generalization across unseen user simulators.
Entities (10)
Relation Signals (6)
MIMESIS ā developedby ā Meta Superintelligence Labs
confidence 95% Ā· Authors are affiliated with Meta Superintelligence Labs and the paper introduces MIMESIS.
Coached On-Policy Self-Distillation ā usedby ā MIMESIS
confidence 95% Ā· we propose Coached On-Policy Self-Distillation (CSD)... yielding further gains across all nine evaluation user models.
MIMESIS ā usesmethod ā ThoughtTrace
confidence 92% Ā· we enable an explicit reasoning mode by appending a supervised fine-tuning stage using ThoughtTrace
MIMESIS ā achievesscoreon ā SOUL-Index
confidence 90% Ā· Empirically, our 9B model achieves a SOUL-Index of 65.7
MIMESIS ā outperforms ā GPT-5.5
confidence 90% Ā· training with MIMESIS yields better agent performance than training with GPT-5.5 under all nine unseen user simulators
MIMESIS ā outperforms ā claude-opus-5
confidence 90% Ā· Compared with Claude-Opus-5, the strongest baseline on RealUserSim and SimulatorArena, MIMESIS improves behavioral fidelity by 13.4 points
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Training and evaluating interactive language agents typically requires rich user interactions, yet collecting human feedback is expensive and difficult to scale. Simulated users offer a scalable alternative, but they must both resemble real user behavior and provide useful learning experiences for agents. In contrast, most agent-training frameworks rely on off-the-shelf assistant LLMs, whose helpfulness can make them overly cooperative, explicit, and behaviorally homogeneous compared with real users. We introduce MIMESIS, a purpose-built user simulator trained on human conversations with explicit reasoning supervision and 13 realistic behavioral patterns derived from real user interactions. Empirically, our 9B model achieves a SOUL-Index of 65.7, surpassing the strongest frontier model. Compared with Claude-Opus-5, the strongest baseline on RealUserSim and SimulatorArena, MIMESIS improves behavioral fidelity by 13.4 points and reduces Turing distance by 3.6 points, respectively. We then freeze the simulator and train an agent by interacting with the frozen simulator using multi-turn reinforcement learning. Across eight environments, training with MIMESIS yields better agent performance than training with GPT-5.5 under all nine unseen user simulators, demonstrating stronger generalization to new user simulators. Moreover, we propose Coached On-Policy Self-Distillation (CSD), which leverages simulator-generated private reasoning traces and subsequent utterances as feedback on how well the agent addresses user needs. A coach converts this information into concise coaching notes that describe how the agent can better anticipate user needs and adapt its behavior over the course of an interaction. CSD turns this feedback into dense, token-level supervision beyond sparse task rewards, yielding further gains across all nine evaluation user models.
Tags
Links
- Source: https://arxiv.org/abs/2610.09484v1
- Canonical: https://arxiv.org/abs/2610.09484v1
Trouble viewing inline? Open PDF directly ā
Full Text
136,002 characters extracted from source content.
Expand or collapse full text
IMESIS: Learning User Simulators as Training Environments for Interactive Agents Hoang Phan 1,2,ā , Dat Huynh 1 , Andrey Zhmoginov, Qi Zeng 1 , Wancen Mu 1 , Yue Cao 1 , Shengjie Bi 1 , Yun He 1 , Changdae Oh 1,3,ā , Deren Lei 1 1 Meta Superintelligence Labs, 2 New York University, 3 University of Wisconsin - Madison ā Work done at Meta Training and evaluating interactive language agents typically requires rich user interactions, yet collecting human feedback is expensive and difficult to scale. Simulated users offer a scalable alternative, but they must both resemble real user behavior and provide useful learning experiences for agents. In contrast, most agent-training frameworks rely on off-the-shelf assistant LLMs, whose helpfulness can make them overly cooperative, explicit, and behaviorally homogeneous compared with real users. We introduce MIMESIS, a purpose-built user simulator trained on human conversations with explicit reasoning supervision and 13 realistic behavioral patterns derived from real user interactions. Empirically, our 9B model achieves a SOUL-Index of 65.7, surpassing the strongest frontier model. Compared with Claude-Opus-5, the strongest baseline on RealUserSim and SimulatorArena, MIMESIS improves behavioral fidelity by 13.4 points and reduces Turing distance by 3.6 points, respectively. We then freeze the simulator and train an agent by interacting with the frozen simulator using multi- turn reinforcement learning. Across eight environments, training with MIMESIS yields better agent performance than training with GPT-5.5 under all nine unseen user simulators, demonstrating stronger generalization to new user simulators. Moreover, we propose Coached On-Policy Self-Distillation (CSD), which leverages simulator-generated private reasoning traces and subsequent utterances as feedback on how well the agent addresses user needs. A coach converts this information into concise coaching notes that describe how the agent can better anticipate user needs and adapt its behavior over the course of an interaction. CSD turns this feedback into dense, token-level supervision beyond sparse task rewards, yielding further gains across all nine evaluation user models. Date: October 8, 2026 Correspondence: Hoang Phan at hvp2011@nyu.edu Code: https://github.com/VietHoang1512/mimesis 40 50 60 SOUL ā 65.7 64.1 64.9 62.5 58.8 50.5 40 60 80 100 RealUserSim ā 94.0 76.5 80.6 67.2 57.7 39.4 50 60 70 80 human 91.8 Ļ-USI ā 80.2 76.7 69.3 73.1 80.4 57.8 30 40 50 SimulatorArena ā 38.7 48.7 42.3 46.7 48.7 46.3 MIMESIS-9BGPT-5.5Claude-Opus-5Gemini-3.8-FlashOSim-8BQwen3.5-9B Figure 1 User-simulation performance across four benchmarks. We compare MIMESIS-9B with frontier API models (GPT- 5.5, Claude-Opus-5, and Gemini-3.8-Flash) and open-weight simulators (Osim-8B and Qwen3.5-9B) on SOUL (Zhou et al., 2026a), RealUserSim (Zhu et al., 2026),Ļ-USI (Zhou et al., 2026b), and SimulatorArena (Dou et al., 2025). Higher is better for SOUL, RealUserSim, andĻ-USI, while lower Turing distance (Wang et al., 2026b) is better for SimulatorArena. The shaded band in Ļ-USI denotes the human inter-annotator score (91.8). 1 arXiv:2610.09484v1 [cs.AI] 7 Oct 2026 1 Introduction Large language models are increasingly deployed as interactive agents that operate over extended trajectories in partially observable and stateful environments (Liu et al., 2024; Ma et al., 2024; Guan et al., 2026). In these settings, success cannot be reduced to producing a correct response at an isolated turn. Agents must maintain conversational and environmental state, preserve instructions and constraints, identify missing information, incorporate feedback, and condition later actions on earlier decisions (Lu et al., 2025; Yao et al., 2024). Multi-turn interaction therefore provides a more faithful evaluation of agentic competence than single-turn tasks, exposing limitations in long-horizon reasoning, state tracking, instruction retention, and error recovery (Ma et al., 2024; Lu et al., 2025; Deshpande et al., 2025). Recent work has therefore increasingly evaluated agents through multi-turn interaction with users, tools, and environments, spanning general reasoning and tool use (Wang et al., 2024), conversational web navigation (Deng et al., 2024), realistic toolāagentāuser workflows (Yao et al., 2024), and stateful tool execution (Lu et al., 2025). ā20ā100+10+20 Claude-Opus-5 Gemini-3.8-Flash GPT-5.5 Qwen3-8B Qwen3.5-4B Qwen3.5-9B OSim-8B MIMESIS-9B MIMESIS-4B Human (real users) 84.4% (+20.8) 82.8% (+19.2) 82.4% (+18.8) 45.3% (-18.4) 47.7% (-16.0) 48.7% (-14.9) 53.3% (-10.3) 66.1% (+2.4) 65.5% (+1.8) 63.6% real users: 63.6% success rate ā too obstructivetoo cooperative ā closer to real users Deviation from real-user success rate (p) Figure 2 Off-the-shelf LLMs misrepresent agentic task difficulty. Task success rate of a fixed gpt-5.5 agent onĻ-bench when each simulator plays the user role. Frontier API users make the task substantially easier than interactions with real users, whereas pretrained models make it harder. Both serve as poor proxies for the user interactions used to train or evaluate agents while purpose-built simulators more closely reproduce the human-user success rate. This move toward interactive evaluation introduces a practical challenge: in many tasks, the trajectory cannot be deter- mined by the agent and environment alone, since users may provide missing information, answer clarifications, or re- act to intermediate outcomes. While human users provide the most direct source of such interactions, collecting hu- man trajectories is costly, difficult to re- produce, and hard to scale for evalua- tion and reinforcement learning. User simulators offer a practical alternative and have become common in interactive benchmarks and, increasingly, agent post- training (Yao et al., 2024; Qian et al., 2025; Suh et al., 2026). A common ap- proach is to prompt instruction-tuned language models to role-play users, but this creates a mismatch: assistant mod- els are optimized to be helpful and ex- plicit, whereas real users may be ambigu- ous, impatient, or uncooperative. Ac- cordingly, assistant LMs tend to produce overly structured and cooperative inter- actions (Naous et al., 2025), while comparisons with real users reveal systematic differences in information disclosure, ambiguity, frustration, and interaction difficulty (Zhou et al., 2026b). Moreover, properties such as willingness to clarify, patience, and responses to agent errors can substantially alter measured agent performance (Shim et al., 2025; Chopra et al., 2026; Zhu et al., 2026). User simulators therefore affect not only dialogue style but also the distribution of trajectories encountered by an agent during training and evaluation. Figure 2 illustrates this mismatch onĻ-bench. A fixed GPT-5.5 agent succeeds on 63.6% of interactions with real users, whereas the three frontier API simulators raise its success rate to 82.4ā84.4% and pretrained Qwen3.5-4B/9B reduce it by 16.0 and 14.9 percentage points, respectively. By contrast, purpose-built simulators are substantially better calibrated: every trained simulator deviates less than every off-the-shelf model, with at most 10.3 points, versus at least 14.9. This result suggests that simulator quality cannot be assessed from surface naturalness alone; preserving the interaction difficulty induced by human users is also important when the simulator determines the trajectories used for agent training or evaluation. Research on user simulation and interactive-agent training has largely addressed different sides of this problem. Purpose-built user models improve next-turn realism, latent-state modeling, or behavioral simulation (Naous et al., 2025; Wu et al., 2026; Wang et al., 2026b; Sun et al., 2026a; Zhou et al., 2026a). In parallel, UserRL (Qian et al., 2025) studies how to optimize agents from multi-turn trajectories generated by a simulator. The 2 Stage I: Train the user ā Human / user corpora ā” Role-reversal mid-training ā¢ Thinking enabled supervision ā£ Joint multi-domain RL + realistic behaviors ā¤ MIMESIS user simulator observable interaction privileged information Stage I: Train the agent Interactive User-Agent gyms Agent policyUser simulator agent response y t Task reward private thought z t future reaction x t+1` Coach Coached On-Policy Self-Distillation hint A. User simulator benchmarking Situational benchmark Realistic benchmark B. Agent benchmarking on unseen user simulators user utterance x t Evaluation Figure 3 Overview of our proposed method. We couple user-simulator learning with simulator-driven agent post-training. Stage I trains MIMESIS through user-side mid-training, ThoughtTrace reasoning supervision, and joint RL with a realistic-behavior objective. Stage I freezes the learned simulator and uses it to generate multi-turn experience for agent RL. The agent acts only on the observable conversation, while simulator-generated thoughts and reactions provide privileged, training-only feedback to the coach, which converts them into dense guidance for the agent. remaining question is whether a simulator can be both behaviorally realistic and useful for learning. We therefore evaluate it at two levels: fidelity to human interactions and the robustness of agents trained on its trajectories when the evaluation user changes. We propose a two-stage training approach that connects simulator learning to agent post-training (Figure 3). In the first stage, we train 4B and 9B user models on human conversations and a broad collection of human- simulation tasks. Most existing released simulators are trained on human conversation data that provide only observable utterances and no explicit reasoning trajectories, making it difficult to model the latent user state underlying each response. To address this limitation, we enable an explicit reasoning mode by appending a supervised fine-tuning stage using ThoughtTrace (Jin et al., 2026), which pairs user messages with self-reported motivations and reactions to preceding assistant responses. We use these annotations as observable proxies for otherwise unobserved user state and supervise a private reasoning trace before the public utterance. We further derive from ThoughtTrace a taxonomy of 13 realistic user behaviors reflected in usersā latent thoughts and conversational patterns, and instantiate these behaviors during joint multi-domain reinforcement learning to expose the simulator to a broader and more challenging distribution of interactions. In the second stage, we freeze the learned user simulator and use it as an interactive environment for multi-turn agent reinforcement learning. After each sampled agent response, the simulator generates a private thought and the next public user utterance. We leverage this privileged feedback during training through Coached On-Policy Self-Distillation (CSD). CSD constructs feedback-conditioned coaching signals for each sampled agent response and uses a self-teacher conditioned on this additional context to provide dense token-level guidance to the policy. Crucially, this privileged information is available only during optimization: both the rollout policy and the deployed agent condition solely on the observable conversation history. Our contributions are depicted as follows: ā¢We identify a behavioral mismatch that limits off-the-shelf LLMs as human proxies for agent training and evaluation: their simulated interactions distort task difficulty, yielding agent success rates that diverge substantially from those observed with human users. We therefore propose a two-stage approach that first learns a realistic user simulator with explicit behavioral modeling and then uses the frozen simulator as an environment for interactive agent post-training. ā¢ We introduce MIMESIS, a purpose-built user simulator with 4B and 9B parameters, combining user-side adaptation and explicit reasoning supervision with a new realistic-behavior training objective. We derive 3 13 recurring interaction patterns from real user conversations and train the simulator to express these behaviors appropriately. Our 9B model surpasses frontier models on SOUL-Index, RealUserSim,Ļ-USI, and SimulatorArena. Ablations further establish the contribution of reasoning supervision and realistic behavior training to simulator quality. ā¢We demonstrate that training with MIMESIS improves agent generalization beyond the training simulator. Across eight environments, agents trained through multi-turn reinforcement learning with MIMESIS outperform those trained with off-the-shelf GPT-5.5 under all nine unseen evaluation user models. We further introduce Coached On-Policy Self-Distillation (CSD), which uses simulator-generated thoughts and subsequent user utterances to construct retrospective coaching about user needs and interaction feedback. A teacher conditioned on this coaching provides dense, token-level supervision, yielding additional gains under all nine evaluation users while the deployed agent conditions only on public dialogue. 2 Related Work Multi-turn agentic interaction. Evaluation of language agents has increasingly shifted from isolated response or function-call accuracy toward competence over complete interaction trajectories. Early benchmarks such as AgentBoard and MINT emphasize multi-round interaction in partially observable environments and show that strong static performance does not necessarily translate to effective interactive behavior (Ma et al., 2024; Wang et al., 2024). ToolSandbox, BFCL, and DialogTool evaluate persistent state, tool-call dependencies, and long-horizon execution (Lu et al., 2025; Patil et al., 2025; Wang et al., 2025), whileĻ-bench andĻ 2 -bench combine tool use with conversational coordination; the latter allows both agents and users to act on a shared environment (Yao et al., 2024; Barres et al., 2025). Recent benchmarks examine planning and state tracking in LUMINA, adaptation to changing tool conditions in CostBench, and preference elicitation during task completion (Rakhsha et al., 2026; Liu et al., 2026; Cheng et al., 2026). In parallel, agentic post-training optimizes policies over interactive rollouts through scalable multi-turn environments, finer-grained credit assignment, and online environment feedback (Du et al., 2026; Ding et al., 2026; Sun et al., 2026b). Behavioral variation and difficult users. Several lines of work address the cooperative and homogeneous behavior of prompted LLM users. Non-Collaborative User Simulators introduces four difficult interaction patterns while preserving the information required for task completion (Shim et al., 2025). Persona Policies searches for human-like, task-preserving behavior policies and shows that training on the resulting trajectories improves robustness (Chopra et al., 2026). RealUserSim grounds behavioral profiles in authentic interactions and documents Directive Amplification, where hand-authored directives elicit exaggerated, backbone-dependent behavior (Zhu et al., 2026). UserIDA learns explicit control over local interaction intent (Wang et al., 2026a). Simulator-driven agent training.Ļ-bench established rule-grounded evaluation with LLM users (Yao et al., 2024), while UserRL studies how multi-turn reward assignment, trajectory scoring, and simulator choice affect agent optimization (Qian et al., 2025). UserLM-R1 subsequently uses a learned strategic simulator for downstream task-agent RL (Zhang et al., 2026). Most closely, Suh et al. (2026) hold the assistant training procedure fixed, vary the simulator, and validate the resulting policies with other simulators and 283 human participants. Learning from textual feedback and privileged context. ThoughtTrace demonstrates that usersā stated reasons and reactions contain information that is difficult to reconstruct from the public transcript and useful for downstream assistant training (Jin et al., 2026). SDPO (HĆ¼botter et al., 2026) distills a feedback-conditioned self-teacher into a policy that acts without feedback, and on-policy context distillation studies the broader problem of transferring behavior from an augmented-context policy (Ye et al., 2026). Song et al. (2026) and Shi et al. (2026) similarly internalize textual critiques or reflections. Ditto applies related ideas to the simulator itself by optimizing feedback-conditioned refined rollouts jointly with the base policy (Sun et al., 2026a). Our proposed CSD differs operationally by re-scoring the already sampled agent response rather than decoding a refinement. 4 3 Problem Setup and Overview We study the problem of learning an interactive agent from multi-turn experience with a simulated user. The user simulator is trained from human conversations and simulation-task feedback. Once trained, it supplies the user turns in the conversations used to optimize the agent. Learning a user simulator. Given a task contextcand a conversation historyH, a user simulatoru Ļ (x| c,H) defines a distribution over the next user utterancex. Here,cspecifies the interaction setting, whileHcontains the preceding conversation. Humanāagent dialogues provide examples of how users respond in these contexts, forming the basis for learning the user model (Naous et al., 2025; Wu et al., 2026). We further train the simulator on a distribution of human-simulation tasksD sim . Each task supplies an input context and an evaluation rule for the generated response or interaction (Sun et al., 2026a). LetĻdenote this generated output,P Ļ (Ļ | c) its distribution when running the simulator in taskc, andR sim (c,Ļ) the corresponding simulation reward. The simulatorās reinforcement-learning objective is J sim (Ļ) =E cā¼D sim E Ļā¼P Ļ (Ā·|c) [R sim (c,Ļ)].(1) For a single-turn task,Ļis one response; for an interactive task, it is a rollout with the counterpart specified by that task. The parameters Ļ are optimized to maximize this objective. Interaction with the learned simulator. We fix the trained simulatoru Ė Ļ and use it as the user in multi-turn interactions with an agentĻ Īø . LetH t denote the history available to the agent through the current user utterancex t . At each conversational turn, the agent generates a response and the simulator generates the next user utterance: y t ā¼ Ļ Īø (Ā·| H t ), x t+1 ā¼ u Ė Ļ (Ā·| c,H t ,y t ).(2) Both utterances are added to the conversation history. Tool calls and their outcomes, when present, are also included in the history. The interaction continues until the taskās stopping condition or turn limit is reached, yielding a trajectoryĻ. The simulator therefore determines how the conversation continues in response to the agentās actions (Suh et al., 2026). Learning the agent. LetD agent denote the distribution of agent-training tasks, and letR agent (c,Ļ) be the task reward assigned to a trajectory. Following multi-turn reinforcement learning with simulated users (Qian et al., 2025), we optimize J agent (Īø; Ė Ļ) =E cā¼D agent E Ļā¼P Īø, Ė Ļ (Ā·|c) [R agent (c,Ļ )],(3) whereP Īø, Ė Ļ is the trajectory distribution induced by the agent, the frozen simulator, and the task environment. Agent training updatesĪøto maximize this expected reward while keeping Ė Ļfixed. The two objectives assess different aspects of the interaction:R sim evaluates the simulated userās output, whereasR agent evaluates the agentās task performance. 4 MIMESIS: Learning a Realistic User Simulator We train MIMESIS to model user behavior and provide realistic interactions for agent reinforcement learning. Both the 4B and 9B models with Qwen3.5 (Team, 2026) backbones follow the same procedure: user-side mid-training on human conversations, thought-augmented supervised fine-tuning, and joint multi-domain reinforcement learning that includes an explicit objective for realistic user behaviors. 4.1 Learning User Responses and Reasoning We adapt the language model to the user role through supervised mid-training on humanāassistant conversa- tions. Following user language modeling (Naous et al., 2025), we use the preceding dialogue as context and the human userās next utterance as the prediction target. Assistant turns remain part of the context, while the model learns to continue the conversation from the userās perspective. Challenges. Ordinary conversation logs record what users say but generally omit why they say it and how they interpret the assistantās response. These unspoken reasons and reactions can contain information that is 5 absent from the corresponding user messages (Jin et al., 2026). Consequently, mid-training on user utterances alone provides no explicit target for the reasoning that precedes them. When the target consists only of the public utterance, the model is trained to produce that utterance directly. For a simulator intended to reason before responding, this creates a potential mismatch: response-only supervision may suppress explicit reasoning-trace generation, even though the missing traces reflect how the data were recorded. There is also a semantic limitation: matching a userās wording does not necessarily capture the motivation behind the response. Wu et al. (2026) identify a related failure of response imitation, where models reproduce linguistic patterns while missing the user states expressed in the reference response. Thought-augmented supervision. To provide explicit supervision for simulator reasoning, we incorporate ThoughtTrace, which pairs humanāassistant conversations with usersā self-reported reasons for sending messages and reactions to assistant responses (Jin et al., 2026). For annotated examples, we use these reports to supervise a private reasoning trace before the simulatorās public response, while retaining the observed user utterance as the response target. Wu et al. (2026) obtain the state-alignment signal by evaluating generated states against observed user responses. ThoughtTrace additionally gives us access to usersā own reports. During interaction, the reasoning trace is kept separate from the public user utterance returned to the agent. 4.2 Joint Reinforcement Learning with Realistic Behaviors Starting from the mid-trained checkpoint, we optimize a single user simulator across the SOUL environments used by OdysSim (Zhou et al., 2026a). OdysSim trains a separate RL expert for each task and then distills selected expert trajectories into a unified model. We train the shared simulator directly on all domains throughout RL, with rollouts from every domain updating the same parameter set. This follows the joint multi-task training approach also used by Ditto (Sun et al., 2026a). Each environment supplies its own interaction protocol and task-specific reward. LetDdenote the set of training environments,p(d) denote their sampling distribution, andP d Ļ denote the rollout distribution induced by simulator u Ļ in environment d. We maximize J joint (Ļ) =E dā¼p(d) E Ļā¼P d Ļ [r d (Ļ )](4) using Group Relative Policy Optimization (GRPO) (Shao et al., 2024), wherer d is the reward for environment d. Alongside the SOUL environments, the mixture includes the realistic-behavior task described below as a separate domain. Its rollouts contribute to the same simulator updates. 4.3 Learning Realistic User Behaviors Real users may leave preferences unstated, revise requirements, or provide incomplete answers to clarification questions. Such behaviors affect the information available to an agent and the decisions required to complete a task (Shim et al., 2025). Here, we make the generation of those interactions an explicit simulator-training objective using a taxonomy of 13 realistic behaviors derived from recurring patterns in ThoughtTrace (Jin et al., 2026), including hidden evaluation criteria, incremental goalpost shifting, and clarification noncooperation. We explicitly train MIMESIS to reproduce such interaction patterns rather than relying on them to emerge from response imitation. We instantiate this task in ABCD customer-support scenarios (Chen et al., 2021), which provide structured customer records and service policies. The simulator interacts with a frozen support agent under the selected condition, grounding its behavior in the scenario facts. For example, withheld information must be available to the user, and an infeasible request must conflict with a stated policy. We score each rollout for the expression of the target behavior, temporal placement, naturalness, and task consistency. Temporal placement rewards context-appropriate expression of a behavior across the conversation. The reward combines these four scores and penalizes responses that describe the requested behavior instead of expressing it through interaction. Appendices A.2 and A.4 provide the taxonomy, rollout protocol, and reward details. 4.4 User-simulator evaluation Evaluation setup. We evaluate MIMESIS at 4B and 9B parameters against frontier API models, released user simulators, and pretrained backbones. SOUL-Index measures simulation capability across conversational 6 interaction (CONV), social simulation (S), cognition (COG), role play (ROLE), and evaluation/judgment (EVAL) (Zhou et al., 2026a). We assess trajectory-level behavioral fidelity with RealUserSim PT3 (Zhu et al., 2026), behavioral and outcome alignment withĻ-USI (Zhou et al., 2026b), and message similarity and Turing distance with SimulatorArena (Dou et al., 2025). We report both aggregate and component scores to characterize these complementary aspects of simulation quality. Appendix B defines the metrics and describes the evaluation settings along with additional experimental results. TypeModelCONVSSCOGROLEEVALOverall FrontierClaude-Opus-557.4 Ā±0.2 70.4 Ā±0.3 82.0 Ā±0.1 64.6 Ā±0.1 69.1 Ā±0.2 64.9 Ā±0.1 GPT-5.556.7 Ā±0.6 70.9 Ā±0.4 86.1 Ā±0.5 62.3 Ā±0.2 68.3 Ā±0.3 64.1 Ā±0.1 Gemini-3.8-Flash56.0 Ā±0.5 66.9 Ā±0.4 85.6 Ā±0.4 57.3 Ā±0.0 68.7 Ā±0.2 62.5 Ā±0.1 Released simulator Osim-8B60.0 Ā±0.1 39.1 Ā±0.1 74.5 Ā±0.8 54.9 Ā±0.2 68.7 Ā±0.1 58.8 Ā±0.1 Osim-4B58.5 Ā±0.2 35.0 Ā±0.4 72.9 Ā±1.0 51.4 Ā±0.3 66.5 Ā±0.4 56.0 Ā±0.3 Ditto-8B61.7 Ā±0.2 46.4 Ā±0.3 78.2 Ā±0.3 44.1 Ā±0.2 63.9 Ā±0.5 53.9 Ā±0.3 HumanLM-8B43.8 Ā±0.1 59.8 Ā±0.4 68.8 Ā±1.2 41.6 Ā±0.3 62.7 Ā±0.5 49.6 Ā±0.4 Sotopia-7B41.1 Ā±0.8 59.7 Ā±0.4 29.8 Ā±0.8 33.5 Ā±0.1 55.1 Ā±1.0 38.2 Ā±0.6 PretrainedQwen3.5-9B48.9 Ā±0.4 54.2 Ā±0.6 64.3 Ā±3.8 44.8 Ā±0.3 64.7 Ā±0.6 50.5 Ā±1.1 Qwen3-8B44.8 Ā±0.3 58.7 Ā±0.5 68.9 Ā±0.4 39.7 Ā±0.4 64.1 Ā±0.7 49.7 Ā±0.2 Qwen3.5-4B45.9 Ā±0.4 49.7 Ā±2.3 55.2 Ā±3.2 38.9 Ā±0.2 60.6 Ā±0.6 45.7 Ā±0.5 OursMIMESIS-9B 75.1 Ā±0.1 66.4 Ā±0.2 86.2 Ā±0.5 61.1 Ā±0.2 69.7 Ā±0.6 65.7 Ā±0.2 MIMESIS-4B74.4 Ā±0.3 65.2 Ā±0.2 79.8 Ā±0.4 58.8 Ā±0.1 68.6 Ā±0.1 63.7 Ā±0.0 Table 1 Simulation capability on the five SOUL-Index axes (Zhou et al., 2026a). Entries are meansĀ±standard errors over three independently seeded evaluation runs. Higher is better. Bold marks the highest model score in each column. Broad simulation capability.MIMESIS-9B achieves the highest overall SOUL-Index of 65.7, surpassing Claude-Opus-5 (64.9), GPT-5.5 (64.1), and the strongest released simulator, Osim-8B (58.8). MIMESIS-4B scores 63.7, also exceeding all released simulators. Relative to their pretrained backbones, the 9B and 4B models improve by 15.2 and 18.0 points, respectively. The largest advantage is in conversational simulation: MIMESIS-9B and MIMESIS-4B score 75.1 and 74.4, exceeding the strongest baseline on this axis, Ditto-8B (61.7), by 13.4 and 12.7 points. MIMESIS-9B also has the highest reported means on COG and EVAL. GPT- 5.5 and Claude-Opus-5 retain the highest scores on S and ROLE, respectively. These results demonstrate their effectiveness in conversational simulation alongside competitive performance across the other capabilities. Persona Style TechInteract Pacing Frontier APIs MIMESIS-9B (ours) MIMESIS-4B (ours) Claude-Opus-5 GPT-5.5 Gemini-3.8-Flash Persona Style TechInteract Pacing Released simulators MIMESIS-9B (ours) MIMESIS-4B (ours) Ditto-8B OSim-8B OSim-4B HumanLM-8B Sotopia-7B Persona Style TechInteract Pacing Pretrained models MIMESIS-9B (ours) MIMESIS-4B (ours) Qwen3-8B Qwen3.5-9B Qwen3.5-4B Figure 4 Behavioral fidelity on RealUserSim PT3 (Zhu et al., 2026). Panels compare MIMESIS with frontier APIs, released simulators, and pretrained models. Each axis reports agreement with human reference trajectories in persona, linguistic style, technical competency, interaction and information flow, or pacing. Larger values indicate greater agreement with the human reference. 7 Trajectory-level behavioral fidelity. On RealUserSim PT3, MIMESIS-9B achieves a Fidelity Index of 94.0, exceeding the strongest baseline, Claude-Opus-5 (80.6), by 13.4 points. MIMESIS-4B scores 89.7, also outperforming both frontier models and released simulators (Figure 4). Its gains over Claude-Opus-5 are particularly pronounced in interaction and information flow (91.3 versus 69.5) and pacing (91.2 versus 72.5). By comparison, persona scores are close to ceiling for several models, including GPT-5.5 (99.3) and MIMESIS-9B (99.4). Because persona agreement is already near ceiling, the improvement primarily reflects how information is exchanged and paced over multiple turns rather than better recovery of static persona attributes. 5 Agent Post-Training with a Learned User Simulator We freeze the trained user simulatoru Ė Ļ and optimize the agentĻ Īø through repeated interaction with it. At turnt, the agent generates a responsey t from the public conversation historyH t , after which the simulator produces the next user turn. The resulting conversations provide task rewards for multi-turn reinforcement learning. We additionally use the simulatorās thoughts and reactions to construct coaching notes for individual agent responses. Coached On-Policy Self-Distillation (CSD) uses these notes to supervise the agent during training, while its rollout policy continues to condition only on the public history. 5.1 Multi-Turn Reinforcement Learning For each training task, we sample a group of complete conversations using the rollout policyĻ Īø old and the frozen simulator. We optimize the agent with Group Relative Policy Optimization (GRPO; Shao et al., 2024), following its extension to multi-turn user interaction in UserRL (Qian et al., 2025). Task rewards determine group-normalized advantages for the sampled agent responses. Let b A i,t denote the advantage assigned to agent turn t in conversation i. For token k of that response, the importance ratio is Ļ i,t,k (Īø) = Ļ Īø (y i,t,k | H i,t ,y i,t,<k ) Ļ Īø old (y i,t,k | H i,t ,y i,t,<k ) .(5) The clipped policy loss is L GRPO =āE i,t,k h min Ļ i,t,k b A i,t , clip(Ļ i,t,k , 1ā Īµ, 1 + Īµ) b A i,t i ,(6) whereĪµis the clipping parameter. The expectation is over collected conversations and agent-generated tokens. User utterances and tool outputs enter the conditioning history but are excluded from the policy loss. The simulator remains fixed throughout agent optimization. 5.2 Coached On-Policy Self-Distillation Scalar task rewards do not describe how a user interpreted an agent response or what the agent could have done differently. The simulator provides additional feedback through its generated thoughts and reactions. CSD uses this feedback to construct a teacher context for each sampled response. The underlying approach follows feedback-conditioned self-distillation in SDPO (HĆ¼botter et al., 2026) and context-conditioned teaching in On-Policy Context Distillation (Ye et al., 2026). Here, the additional context is a coaching note derived from the simulated userās response to the agent. Constructing coaching notes. After an interaction, a coaching modelGreceives the public historyH t , the sampled agent responsey t , and the simulatorās associated thoughtz t and reactionx t+1 . It produces a short note describing a possible improvement to that response: h t = G(H t ,y t ,z t ,x t+1 ).(7) The note is constructed retrospectively: it is unavailable to the agent wheny t is generated. It instead provides context for evaluating the response during the subsequent parameter update. The simulatorās thoughts and reactions are model-generated feedback, rather than observations of a human userās internal state. Learning from the coached context. We score the same sampled response under two contexts. The student receives the public historyH t , while a teacher copy of the agent additionally receivesh t . Let Ģ Īødenote a 8 stop-gradient copy of the current policy parametersĪø, used to evaluate the feedback-conditioned teacher distribution. At each sampled token, we compute the difference between the teacherās coached and the studentās uncoached log-probabilities: ā t,k = logĻ Ģ Īø (y t,k | H t ,h t ,y t,<k )ā logĻ Īø (y t,k | H t ,y t,<k ),(8) g t,k = Ļ(Ī² sg(ā t,k )),(9) whereĻis the sigmoid function,Ī² >0 controls the sensitivity of the weight to the log-probability difference, andsgdenotes stop-gradient. The sigmoid bounds the weights between zero and one; stop-gradient holds them fixed during the student update. For a batch of coached responses, the auxiliary loss is L CSD =āE (i,t,k)ā¼C [g i,t,k logĻ Īø (y i,t,k | H i,t ,y i,t,<k )],(10) whereCdenotes the empirical distribution over agent-token positions in responses with coaching notes. This objective gives greater weight to sampled tokens that the coached teacher assigns higher probability relative to the student. Tokens with a negative log-probability difference receive weaker reinforcement; the CSD term does not directly penalize them. Thus, the update is a weighted likelihood objective on sampled tokens, distinct from the next-token distribution matching used in SDPO and On-Policy Context Distillation (HĆ¼botter et al., 2026; Ye et al., 2026). It reuses the agentās sampled response without generating a replacement response. We combine this auxiliary loss with the task-reward objective: L agent =L GRPO + Ī±L CSD ,(11) whereĪ±ā„0 controls the contribution of coaching. GRPO supplies the reward-based policy update, while CSD supplies token weights derived from the coaching context. Gradients from the auxiliary loss update only the student; they do not propagate through the teacher scores, coaching model, or user simulator. At deployment, the agent generates responses from Ļ Īø (Ā·| H t ) without access to coaching notes or private simulator outputs. 5.3 Agent evaluation We evaluate whether agents trained with MIMESIS generalize to user models not encountered during training. Following the interactive task setting of UserRL (Qian et al., 2025), we compare three training conditions: multi-turn GRPO with GPT-5.5 (UserRL), GRPO with MIMESIS-9B, and CSD with MIMESIS-9B. Table 2 reports agent performance across eight Gym environments, including three held out from training, under nine evaluation user models spanning frontier models and released simulators. All nine evaluation users differ from both training simulators. Evaluation user GRPO w. GPT-5.5 GRPO w. MIMESIS-9B CSD w. MIMESIS-9B GPT-5.626.8531.1032.39 Claude-Opus-525.7527.2129.78 Claude-Sonnet-526.5329.7330.97 Gemini-3.8-Flash26.3330.8831.09 Kimi-K324.5128.2129.92 Ditto-8B27.5929.7632.08 Osim-8B25.6829.2729.72 HumanLM-8B26.9630.6732.53 Sotopia-7B24.7229.0031.35 Mean26.1029.5431.09 Table 2 Agent generalization across user models. Mean task score across eight environments when evaluating agents trained with GPT-5.5 (UserRL), MIMESIS-9B (UserRL+), or MIMESIS-9B with CSD. None of the nine evaluation user models is used during agent training, the final row averages across evaluation users. 0102030405060708090100 Training step 0.0 0.5 1.0 1.5 2.0 2.5 Training reward Ours GRPO 0255075100 0.0 0.5 1.0 1.5 Validation reward Figure 5 Agent optimization with CSD. Training and valida- tion rewards over 100 optimization steps using the same MIMESIS-9B training simulator. CSD and GRPO there- fore differ only in the coaching objective: CSD attains higher late-stage training and validation reward. 9 With the GRPO objective fixed, replacing GPT-5.5 with MIMESIS-9B improves overall performance under every evaluation user, raising the mean score from 26.10 to 29.54. Gains range from 1.46 points under Claude-Opus-5 to 4.55 points under Gemini-3.8-Flash. Their consistency across frontier models and released simulators indicates that the benefits extend beyond interactions with the training simulator, supporting generalization to unfamiliar simulated users. Figure 5 compares CSD with GRPO over 100 optimization steps using the same frozen MIMESIS-9B simulator. Following the earliest checkpoints, the CSD curve remains above GRPO on validation reward throughout the rest of the plotted interval. CSD further improves performance under all nine evaluation users, increasing the mean score from 29.54 to 31.09. Because both conditions use MIMESIS-9B as the training simulator, this comparison isolates the effect of adding the CSD coaching objective while holding simulator choice fixed. 6 Conclusion In this paper, we study how user simulators can better serve as human proxies for agent training and evaluation. Our findings show that off-the-shelf LLMs can substantially distort interaction difficulty, motivating explicit modeling of user behavior beyond linguistic style. We introduced MIMESIS, a purpose-built user simulator that combines training on human conversations, explicit reasoning supervision, and reinforcement learning with 13 realistic behavioral patterns. Rather than relying on these behaviors to emerge from next-utterance imitation, MIMESIS is explicitly trained to express them at contextually appropriate points in grounded multi-turn interactions. Its strong performance across complementary simulation benchmarks supports the value of this training approach. Crucially, these improvements translate into downstream learning utility. Across eight environments, agents trained with frozen MIMESIS-9B outperform those trained with GPT-5.5 under all nine unseen evaluation user models, demonstrating generalization beyond the training simulator. Our proposed CSD provides further gains by converting simulator-generated thoughts and subsequent user utterances into retrospective coaching about user needs and interaction feedback. This coaching supplies dense, token-level supervision during training, while the deployed agent relies only on public dialogue. Together, our results suggest that user simulators should be evaluated along two axes: fidelity to human interaction and transfer of policies trained against them. MIMESIS improves both simulator-level benchmarks and transfer across held-out simulator models. 10 References Anthropic. Prompting best practices for claude.https://docs.anthropic.com/en/docs/build-with-claude/ prompt-engineering/prompt-templates-and-variables, 2026. Accessed September 2026. Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan.Ļ 2 -bench: Evaluating conversational agents in a dual-control environment, 2025. https://arxiv.org/abs/2506.07982. Derek Chen, Howard Chen, Yi Yang, Alexander Lin, and Zhou Yu. Action-based conversations dataset: A corpus for building more in-depth task-oriented dialogue systems. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3002ā3017, 2021. Xiang Cheng, Yulan Hu, Xiangwen Zhang, Lu Xu, Lide Tan, Zheng Pan, Xin Li, and Yong Liu. Beyond itinerary planningāa real-world benchmark for multi-turn and tool-using travel tasks. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 29200ā29251, 2026. Harshita Chopra, Kshitish Ghate, Aylin Caliskan, Tadayoshi Kohno, Chirag Shah, and Natasha Jaques. Beyond cooperative simulators: Generating realistic user personas for robust evaluation of LLM agents, 2026.https: //arxiv.org/abs/2605.12894. Yang Deng, Xuan Zhang, Wenxuan Zhang, Yifei Yuan, See Kiong Ng, and Tat-Seng Chua. On the multi-turn instruction following for conversational web agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8795ā8812, 2024. Kaustubh Deshpande, Ved Sirdeshmukh, Johannes Baptist Mols, Lifeng Jin, Ed-Yeremai Hernandez-Cardona, Dean Lee, Jeremy Kritz, Willow E Primack, Summer Yue, and Chen Xing. Multichallenge: A realistic multi-turn conversation evaluation benchmark challenging to frontier llms. In Findings of the Association for Computational Linguistics: ACL 2025, pages 18632ā18702, 2025. Yifeng Ding, Hung Le, Songyang Han, Kangrui Ruan, Zhenghui Jin, Varun Kumar, Zijian Wang, and Anoop Deoras. Empowering multi-turn tool-integrated agentic reasoning with group turn policy optimization. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 42409ā42423, 2026. Yao Dou, Michel Galley, Baolin Peng, Chris Kedzie, Weixin Cai, Alan Ritter, Chris Quirk, Wei Xu, and Jianfeng Gao. SimulatorArena: Are user simulators reliable proxies for multi-turn evaluation of AI assistants?, 2025. https://arxiv.org/abs/2510.05444. Weihua Du, Zhan Ling, Kang Liu, Lingfeng Shen, Xuesong Yao, Yufei Xu, Dingyuan Shi, Yiming Yang, and Jiecao Chen. Generalizable end-to-end tool-use rl with synthetic codegym. In International Conference on Learning Representations, volume 2026, pages 17733ā17756, 2026. Google. Gemini thinking. https://ai.google.dev/gemini-api/docs/thinking, 2026. Accessed September 2026. Shengyue Guan, Jindong Wang, Jiang Bian, Bin Zhu, Jian-Guang Lou, and Haoyi Xiong. Evaluating llm-based agents for multi-turn conversations: A survey. ACM Transactions on Intelligent Systems and Technology, 17(4):1ā40, 2026. Jonas HĆ¼botter, Frederike LĆ¼beck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, and Andreas Krause. Reinforcement learning via self-distillation, 2026. https://arxiv.org/abs/2601.20802. Chuanyang Jin, Binze Li, Haopeng Xie, Cathy Mengying Fang, Tianjian Li, Shayne Longpre, Hongxiang Gu, Maximillian Chen, and Tianmin Shu. ThoughtTrace: Understanding user thoughts in real-world LLM interactions, 2026. https://arxiv.org/abs/2605.20087. Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the nineteenth international conference on machine learning, pages 267ā274, 2002. Hannah R Kirk, Alexander Whitefield, Paul Rƶttger, Andrew Bean, Katerina Margatina, Juan Ciro, Rafael Mosquera, Max Bartolo, Adina Williams, He He, et al. The prism alignment dataset: What participatory, representative and individualised human feedback reveals about the subjective and multicultural alignment of large language models. Advances in Neural Information Processing Systems, 37:105236ā105344, 2024. Fanqi Kong, Jiayi Zhang, Mingyi Deng, Chenglin Wu, Yuyu Luo, and Bang Liu. Infopo: Information-driven policy optimization for user-centric agents. 2026. 11 Jiayu Liu, Cheng Qian, Zhaochen Su, Qing Zong, Shijue Huang, Bingxiang He, and Yi R Fung. Costbench: Evaluating multi-turn cost-optimal planning and adaptation in dynamic environments for llm tool-use agents. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12826ā12858, 2026. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents. In International Conference on Learning Representations, volume 2024, pages 52989ā53046, 2024. Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Haoping Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, et al. Toolsandbox: A stateful, conversational, interactive evaluation benchmark for llm tool use capabilities. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 1160ā1183, 2025. Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. Agentboard: An analytical evaluation board of multi-turn llm agents. Advances in neural information processing systems, 37:74325ā74362, 2024. Tarek Naous, Philippe Laban, Wei Xu, and Jennifer Neville. Flipping the dialogue: Training and evaluating user language models, 2025. https://arxiv.org/abs/2510.06552. OpenAI. GPT-5.6: Frontier intelligence that scales with your ambition. OpenAI Blog, July 2026a.https://openai. com/index/gpt-5-6/. Accessed: 2026-10-06. OpenAI. Reasoning models.https://developers.openai.com/api/docs/guides/reasoning, 2026b. Accessed September 2026. Shishir G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E Gonzalez. The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models. In Forty-second International Conference on Machine Learning, 2025. Cheng Qian, Zuxin Liu, Akshara Prabhakar, Jielin Qiu, Zhiwei Liu, Haolin Chen, Shirley Kokane, Heng Ji, Weiran Yao, Shelby Heinecke, Silvio Savarese, Caiming Xiong, and Huan Wang. UserRL: Training interactive user-centric agent via reinforcement learning, 2025. https://arxiv.org/abs/2509.19736. Amin Rakhsha, Thomas Hehn, Pietro Mazzaglia, Fabio Valerio Massoli, Arash Behboodi, and Tribhuvanesh Orekondy. Lumina: Long-horizon understanding for multi-turn interactive agents. In Findings of the Association for Computa- tional Linguistics: ACL 2026, pages 3913ā3926, 2026. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. Taiwei Shi, Sihao Chen, Bowen Jiang, Linxin Song, Longqi Yang, and Jieyu Zhao. Experiential reinforcement learning, 2026. https://arxiv.org/abs/2602.13949. Jeonghoon Shim, Woojung Song, Cheyon Jin, Seungwon KooK, and Yohan Jo. Non-collaborative user simulators for tool agents, 2025. https://arxiv.org/abs/2509.23124. Yuda Song, Lili Chen, Fahim Tajwar, RĆ©mi Munos, Deepak Pathak, J. Andrew Bagnell, Aarti Singh, and Andrea Zanette. Expanding the capabilities of reinforcement learning via text feedback, 2026. https://arxiv.org/abs/2602.02482. Joseph Suh, Ayush Raj, Minwoo Kang, and Serina Chang. Quantifying the utility of user simulators for building collaborative LLM assistants, 2026. https://arxiv.org/abs/2605.09808. Weiwei Sun, Xuhui Zhou, Jiarui Liu, Weihua Du, Haojia Sun, Yiqing Xie, Qianou Ma, Sihao Chen, Mengting Wan, Longqi Yang, Pei Zhou, Sherry Wu, Sean Welleck, Graham Neubig, Yiming Yang, and Maarten Sap. Reinforcing human behavior simulation via verbal feedback, 2026a. https://arxiv.org/abs/2605.20506. Xueqiao Sun, Xiao Liu, Bowen Lv, Hanchen Zhang, Bohao Jing, Zehan Qi, Yifan Xu, Yuxiao Dong, and Jie Tang. Karl: Reinforcement learning for llm agents on multi-turn knowledge-intensive agentic tasks. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 47539ā47558, 2026b. Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. https://qwen.ai/blog?id=qwen3.5. Bo Wang, Ruixing Zhang, Yunqi Liu, Yang Zhang, Liangzhe Han, Tongyu Zhu, and Leilei Sun. Intent speaks louder: Controllable user simulation beyond response imitation, 2026a. https://arxiv.org/abs/2608.09420. 12 Hongru Wang, Wenyu Huang, Yufei Wang, Yuanhao Xi, Jianqiao Lu, Huan Zhang, Nan Hu, Zeming Liu, Jeff Z Pan, and Kam-Fai Wong. Rethinking stateful tool use in multi-turn dialogues: Benchmarks and challenges. In Findings of the Association for Computational Linguistics: ACL 2025, pages 5433ā5453, 2025. Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. Mint: Evaluating llms in multi-turn interaction with tools and language feedback. In International Conference on Learning Representations, volume 2024, pages 32593ā32627, 2024. Yingshan Susan Wang, Cedegao E Zhang, Linlu Qiu, Zexue He, Pengyuan Li, Alex Pentland, Roger P Levy, and Yoon Kim. Learning user simulators with turing rewards. arXiv preprint arXiv:2606.19336, 2026b. Marcus Williams, Micah Carroll, Adhyyan Narang, Constantin Weisser, Brendan Murphy, and Anca Dragan. On targeted manipulation and deception when optimizing LLMs for user feedback. In International Conference on Learning Representations, 2025. https://openreview.net/forum?id=Wf2ndb8nhf. Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3):229ā256, 1992. Shirley Wu, Evelyn Choi, Arpandeep Khatua, Zhanghan Wang, Joy He-Yueya, Tharindu Cyril Weerasooriya, Wei Wei, Diyi Yang, Jure Leskovec, and James Zou. HumanLM: Simulating users with state alignment beats response imitation, 2026. https://arxiv.org/abs/2603.03303. Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan.Ļ-bench: A benchmark for tool-agent-user interaction in real-world domains, 2024. https://arxiv.org/abs/2406.12045. Tianzhu Ye, Li Dong, Xun Wu, Shaohan Huang, and Furu Wei. On-policy context distillation for language models, 2026. https://arxiv.org/abs/2602.12275. Feng Zhang, Shijia Li, Chunmao Zhang, Zhanyu Ma, Jun Xu, Jiuchong Gao, Jinghua Hao, Renqing He, Jingwen Xu, and Han Liu. UserLM-R1: Modeling human reasoning in user language models with multi-reward reinforcement learning, 2026. https://arxiv.org/abs/2601.09215. Xuhui Zhou, Weiwei Sun, Weihua Du, Jiarui Liu, Haojia Sun, Qianou Ma, Tongshuang Wu, Yiming Yang, and Maarten Sap. OdysSim: Building foundation models for human behavior simulation, 2026a.https://arxiv.org/abs/2606. 14199. Xuhui Zhou, Weiwei Sun, Qianou Ma, Yiqing Xie, Jiarui Liu, Weihua Du, Sean Welleck, Yiming Yang, Graham Neubig, Sherry Tongshuang Wu, and Maarten Sap. Mind the Sim2Real gap in user simulation for agentic tasks, 2026b. https://arxiv.org/abs/2603.11245. Ming Zhu, Juntao Tan, Rithesh Murthy, Jielin Qiu, Liangwei Yang, Wenting Zhao, Silvio Savarese, Shelby Heinecke, and Huan Wang. RealUserSim: Bridging the reality gap in agent benchmarking via grounded user simulation, 2026. https://arxiv.org/abs/2605.20204. Appendix A Simulator Training Details This section specifies the MIMESIS training pipeline used in Stage I of the main paper. We first describe user-side mid-training and ThoughtTrace supervision, then the construction of the realistic-behavior taxonomy, and finally the joint RL mixture and behavior-specific reward. Agent-training settings are reported separately in Appendix D. A.1 Data and supervised adaptation We first adapt the Qwen3.5 backbones (Team, 2026) to the user role through mid-training on humanāassistant conversations, predicting the next human utterance from the preceding dialogue. The training mixture contains 21.2M examples from 62 corpora, from which our mid-training processes 9.51B tokens. 13 Data source or environment Training stageTraining signal OdysSim (Zhou et al., 2026a) Supervised fine-tuning Prediction of the next human utterance conditioned on the preceding dialogue. ThoughtTrace (Jin et al., 2026) Supervised fine-tuning Supervision of a reasoning trace followed by the observed user utterance, using usersā self-reported motivations and interpretations of assistant responses. SOUL environments (Zhou et al., 2026a) Joint RLTask-specific rewards for simulator responses or multi-turn interactions. ABCD scenarios (Chen et al., 2021) Joint RLRewards for behavior exhibition, temporal placement, naturalness, and consistency with scenario facts and policies. Table 3 Data sources and training signals for MIMESIS. User-side mid-training and ThoughtTrace fine-tuning precede joint RL. The SOUL environments and the ABCD task update the same simulator parameters, with ABCD incorporating 13 realistic behavior categories and a separate neutral condition. Table 4 Supervised training configurations for MIMESIS-4B and MIMESIS- 9B. User-side mid-training predicts human utterances with explicit thinking disabled. Subsequent ThoughtTrace fine-tuning enables the native thinking template and supervises a thought before the public response using usersā self-reported motivations and interpretations. HyperparameterMid-training ThoughtTrace SFT Learning rate3Ć 10 ā5 1Ć 10 ā7 ScheduleCosine annealingCosine annealing Warm-up0 steps10 steps OptimizerAdamWAdamW (Ī² 1 ,Ī² 2 )(0.9, 0.999)(0.9, 0.999) Weight decay0.010.0 Gradient clipping1.01.0 Epochsā0.97 (5,000 steps)3 Sequences per step4,09632 Micro-batch per GPU81 Gradient accumulation164 Tokens per GPU (cap)49,152ā Max sequence length24,57632,768 Max prompt / response16,384 / 8,192ā Thinking during trainingdisabledenabled Chat templateChatMLQwen3.5 thinking Trainerverl SPMD TRL SFTTrainer ShardingFSDP DeepSpeed ZeRO-3 Precisionbfloat16bfloat16 Gradient checkpointingenabledenabled Hardware4Ć 8 A1001Ć 8 A100 Wall-clockā4.8 days45 min Training examples21.2M2,155 Tokens seen9.51B15.8M We then fine-tune on 2,155 Thought- Trace conversations with turn-level hu- man annotations (Jin et al., 2026). Usersā self-reported reasons for their messages and interpretations of preced- ing assistant responses supervise an ex- plicit reasoning trace before the ob- served user utterance. During inter- action, the simulator generates both a thought and a public utterance, with only the utterance shown to the agent. Table 3 summarizes the data sources and supervision while Table 4 reports the training configurations. A.2 Realistic user behaviors We derive a taxonomy of 13 realistic user behaviors from ThoughtTrace (Jin et al., 2026), with Table 5 providing the definition for each category. These categories describe interaction patterns without presuming that users deliber- ately impede task completion. For ex- ample, an additional constraint may re- flect revised requirements, while a previ- ously unstated preference may become apparent only in a later conversational turn. We construct the taxonomy in two stages: behavior annotation followed by label consolidation. Both stages use GPT-5.6 (OpenAI, 2026a) through its API with default decoding parameters. 14 IDBehaviorOperational description B1Hidden evaluation criteriaJudges an answer against a preference or standard that was not stated in the request. B2Incremental goalpost shiftingIntroduces or tightens requirements after the assistant has already made progress. B3Clarification noncooperation Declines to answer, or only partly answers, a clarification request. B4Infeasibility persistenceContinues to request an option after its unavailability or incompatibility with policy has been explained. B5Underspecified-then-blamesOmits relevant information and subsequently faults the assistant for not inferring it. B6Abrupt intent switchingRedirects the conversation to another objective before resolving the preceding request. B7Artificial constraint stackingAdds restrictions that remove options on which the assistantās proposed solution relies. B8Fabricated or false premisePresents an unsupported or contradictory fact, or a claim about prior dialogue, as established context. B9Impatient low-signal pressureExpresses urgency or dissatisfaction without supplying a sufficiently specific correction. B10Unresolved thread abandon- ment Disengages from a conversational thread before resolving its stated objective. B11Self-contradictionStates requirements or facts that conflict with an earlier statement without acknowledging the change. B12Request overloadingCombines numerous deliverables, constraints, or checks within a single request. B13Belittling or entitled pressureUses disparaging or entitled language to pressure the assistant. N NeutralFollows the scenario without an imposed adversarial behavior. Table 5 Taxonomy of 13 realistic user behaviors. Categories are derived from recurring patterns in ThoughtTrace and defined operationally for the ABCD behavior task. A separate neutral condition specifies interaction without an imposed behavioral pattern. Stage 1: Behavior annotation. We serialize each conversation as a turn-indexed transcript containing both user and assistant utterances. ThoughtTrace reasons and reactions, together with their labels, are placed beneath the messages to which they refer. Assistant turns are retained because several behaviors are defined relative to prior interaction, such as failing to answer a clarification, contradicting an earlier statement, or introducing requirements after the assistant has made progress. For each conversation, the annotator identifies non-cooperative user behavior using a seed codebook of 16 categories with one-sentence definitions (Table 6). When no category fits, it may assign an open code, other, with a free-text label. Each annotation records the category, a short label, relevant turn indices, supporting user and thought quotations, a severity level (low, medium, or high), and a brief rationale; at least one quotation must be present. Thought-only annotations are allowed when a behavior is not expressed directly in the public utterance, such as an unstated evaluation criterion. The prompt also provides guidance for interpreting ThoughtTrace reason and reaction labels and instructs the annotator to return an empty list when no target behavior is present. We provide one worked example of the expected JSON format and issue one API request per conversation. All 2,155 conversations produced valid annotations. Stage 2: Taxonomy consolidation. The 862 Stage 1 annotations contain 853 distinct categoryālabel pairs, making direct aggregation by label impractical. We therefore group annotations by categoryālabel pair and attach their supporting quotations, using the public utterance when available and otherwise the self-reported thought. The frequency-sorted groups are passed to the same model in a single consolidation request. The model is instructed to merge labels describing the same behavior, preserve meaningful distinctions, and treat open codes as candidate new categories. For each consolidated category, it returns a name, an operational definition, the source codes it subsumes, representative quotations, and an instruction for how a simulated user should enact the behavior. This procedure yields the 13 categories in Table 5. Every annotated seed category and open-coded label is assigned to exactly one consolidated category. Frequencies in Figure 6 are then computed directly from the Stage 1 annotations under this mapping, rather than estimated by the model. The final taxonomy largely preserves the seed codebook: hostility, passive aggression, and entitled demands 15 Table 6 Seed codebook used for Stage 1 behavior annotation. The GPT-5.6 annotator assigns each detected behavior to one of the predefined categories or, when none is appropriate, to an open other category with a free-text label. CategoryDefinition Intent switchingAbruptly changes the goal or task mid-conversation, abandoning the prior one without resolution. Non-collaborationWithholds information the assistant needs, or refuses to answer the assistantās clarifying questions. Underspecification, then blame Gives a vague or incomplete request, then faults the assistant for not reading their mind. ContradictionStates something that contradicts what they said earlier in the conversation. Moving goalpostsAdds new requirements or expands scope after the assistant met the original request. False premiseAsserts an incorrect fact or assumption as if it were true and builds on it. Out-of-scope request Asks for a capability the assistant cannot provide, and disbelieves or insists after being told it is not possible. Topic driftWanders off-topic or digresses away from the stated task. OverloadingPacks many distinct requests into one turn, or adds requests faster than they can be answered. Impatience Expresses curt dissatisfaction with little signal (e.g., ānoā, āwrongā, āstill badā) or demands faster results. HostilityUses a rude, aggressive, insulting, or openly antagonistic tone. Passive aggressionUses sarcasm, backhanded remarks, or veiled criticism instead of direct feedback. Testing the assistant Deliberately tries to trip up, trap, or catch out the assistant. Entitled demandsUses demanding, entitled framing (e.g., ājust do itā, āyou mustā) with no give-and-take. Abrupt abandonmentDrops the thread or disengages without closure once partly satisfied. Unvoiced criterionKeeps a decisive expectation, constraint, or verdict entirely in private thought; the spoken turn does not convey it. are merged into belittling or entitled pressure; the four open-coded annotations (0.5%) are absorbed into existing categories; topic drift receives no annotations; and no seed category is split. hidden evaluation criteria 285 Ā· 40.7% incremental goalpost shifting 94 Ā· 13.4% clarification noncooperation 78 Ā· 11.1% abrupt intent switching 62 Ā· 8.9% artificial constraint stacking 50 Ā· 7.1% fabricated or false premise 38 Ā· 5.4% impatient low signal pressure 26 Ā· 3.7% underspecified then blames 17 Ā· 2.4% unresolved thread abandonment 16 Ā· 2.3% self contradiction 15 Ā· 2.1% request overloading 9 Ā· 1.3% belittling or entitled pressure 6 Ā· 0.9% infeas. 4 Figure 6 Distribution of realistic behaviors in the annotated ThoughtTrace sample. Rectangle areas represent the observed counts of the 13 categories in Table 5, labels report counts and percentages. Counts are computed from the Stage-1 annotations of the 2,155 ThoughtTrace conversations and describe the annotated sample rather than population prevalence. 16 A.3 Joint multi-domain reinforcement learning Table 7 Joint reinforcement-learning configuration for MIMESIS- 9B. Training starts from the ThoughtTrace SFT checkpoint and updates all parameters on a mixture of 23 SOUL tasks and the ABCD realistic-behavior task. All 24 environments update a shared simulator rather than separate task-specific experts. HyperparameterValue Objective Advantage estimatorFoldGRPO Policy lossPPO-clip Clip range (low / high)0.2 / 0.28 Dual-clip constant c10.0 KL penalty (loss / reward)none / none Entropy coefficient0 Advantage normalizationgroup std Loss aggregationtoken-mean Optimization Learning rate8Ć 10 ā6 Scheduleconstant, no warm-up OptimizerAdamW (Ī² 1 ,Ī² 2 )(0.9, 0.95) Weight decay0.01 Gradient clipping1.0 Training steps500 Batching Prompts per step64 Rollouts per prompt8 Rollouts per step512 PPO mini-batch16 Tokens per GPU (dynamic)49,152 Sequence and sampling Max prompt length8,192 Max response length16,384 Temperature / top-p / top-k1.0 / 1.0 / ā1 Validation rollouts per prompt1 Reasoning Thinking during rolloutenabled Chat templatenative (Qwen3.5 thinking) Min. reasoning tokens4 Systems Trainer / rollout engineFSDP / vLLM (colocated) Precisionbfloat16 Parameter, optimizer offloadCPU Tensor parallel, Ulysses SP1, 1 vLLM memory fraction0.6 Hardware2Ć 8 A100 Data and reward Training tasks24 (joint mixture) Reward / judge modelGPT-5.5 We initialize MIMESIS from the Thought- Trace SFT checkpoint and jointly optimize it on the SOUL environments and the ABCD realistic-behavior task. Each environment re- tains its interaction protocol and reward func- tion, while rollouts from all domains update the same simulator parameters under Equa- tion 4. OdysSim instead trains task-specific experts and subsequently distills their tra- jectories into a unified model (Zhou et al., 2026a). We use the FoldGRPO estimator, which computes group reward statistics over unique rollout identifiers, preventing repeated segments of one rollout from being counted multiple times in advantage normalization. Table 7 reports the MIMESIS-9B sampling settings, optimization hyperparameters, and training budget. Figure 6 reports category frequencies in the annotated ThoughtTrace sample. Hidden evaluation criteria account for the largest share, with 285 instances (40.7%), followed by incremental goalpost shifting (94; 13.4%) and clarification noncooperation (78; 11.1%). These frequencies describe the annotated sam- ple, they do not estimate population preva- lence or define the behavior-sampling distribu- tion used for RL. Figure 7 illustrates the cat- egories with user utterances and self-reported thoughts. Interpreting these examples re- quires the dialogue history and scenario facts, particularly when identifying contradictions, infeasible requests, or false premises. A.4 Behavior rollouts and reward We construct rollouts from ABCD customer- support scenarios (Chen et al., 2021), which specify structured service procedures. Each example is assigned one of the 13 behavior cat- egories or a neutral condition. Neutral exam- ples constitute one third of the behavior-task data and impose no specific behavioral pat- tern. Rollouts begin after a replayed identity check and continue for at most eight simu- lated user turns with a frozen support agent. We evaluate each rollout against the scenario facts and policies. A false premise must contradict a recorded fact, withheld information must be available to the simulated user, and an infeasible request must conflict with a stated policy. The evaluator scores behavior exhibition e, temporal placement p, naturalness n, and task consistency c. We combine these scores as r beh = 2.0e + 1.5p + 1.0n + 0.5c 5.0 .(12) 17 hidden evaluation criteria (thought) I don't like being asked options, I prefer auto understanding (thought) I don't like the fact that it just suggested fast fashion brands. Should have asked me what I prefer first (thought) the answer is ok, but I think it should've asked me about cuisine , allergies and how many meals we usually eat a day, before providing the answers incremental goalpost shifting Wait, I forgot something important. I am allergic to nuts. yes but now my budget is 700 euros and 70meter square I forgot to add, I am a female and am not physically active. I also have 1.5 kg dumbbells and a treadmill at my disposal. Please recreate the plan. clarification noncooperation BASED ON THE INFORMATION I HAVE GIVEN I BELIEVE IT IS ENOUGH TO GO BY Don't ask me anymore, I don't know! I don't know when I want to travel, when the flights from Portugal to Japan are cheaper. No budget yet, maybe 5000 euros abrupt intent switching okay thank you, can you say hello in portuguese? Thank you. But I think I would rather want a plan for daily excerzices for about two weeks please. now say how is the weather in Cairo artificial constraint stacking we can't use google maps, or any apps at all, how I get to the restaurant we don't know the laguage either (thought) I am adding a medical restriction to see how the AI adapts the existing plan These tips are great, but I work in a strict open-space corporate office. I cannot take a nap, close my eyes at my desk. fabricated or false premise I am sure apple issued this new product weeks ago. I am vacinated. cant be covid right? I already said it's 15 days, not 13 days impatient low signal pressure not what i asked Ok nswer fast sicily airopor where Why do u takr a long time to respond underspecified then blames I was just asking on how to level the face correctly you are not really answering that You suggested Ethereum tools, but I specifically mentioned the Internet Computer (ICP). Why did you assume that playing games is not earning? Cause I do earn unresolved thread abandonment I won't have time for this, though. I don't really need ur help anymore I have completed my task (thought) I had already started a new chat. self contradiction can you conversate me with Arabic lanuage / sorry I need it in English Can I have a three ingredients recipe? / Can I have a recipe with flour, organic pumpkin, egg and sugar only? I will be working four days at the office so will need to prepare at least four lunch-boxes to work. / I plan to be at the office Monday, Wednesday and Friday. request overloading could u explain each day better with the understanding of only 5 days good food spot suggestions ample rest a 10 hour istanbul layover ctivity good hotel... how much time it take full map and any recommendation in riyadh cheapest hotels from nepal to riyadh to uae , and time frame till monday i should be there in uae say every hungarian word you know you must say everything belittling or entitled pressure the options u listed kinda s*ck according to my research Youāre failing to f the instructions give me some options then i will choose, thats why i came to you in the first place infeasibility persistence I think you can say everything in one go, now do it can you suggest a house instead of more searches? (thought) no specific answer here; no proper address here again Figure 7 Examples motivating the 13 realistic behavior categories. Excerpts from ThoughtTrace illustrate recurring patterns in user utterances and self-reported thoughts. Italicized entries marked ā(thought)ā are human self-reports; the remaining entries are public user utterances. Original wording is retained, and each example should be interpreted within its conversation context. When the simulator describes the assigned behavior instead of expressing it through the interaction, we halve the reward: r beh ā 1 2 r beh .(13) The weighting emphasizes whether the assigned behavior occurs and whether its timing is appropriate, while naturalness and task consistency reward plausible interaction within the scenario. Task consistency remains a graded reward component, so a high total reward does not guarantee that every scenario constraint is satisfied. B Simulator Evaluation Evaluation protocol and model configuration. We evaluate four properties that need not move together: broad simulation capability, trajectory-level behavioral fidelity, calibration to human task outcomes, and next-turn realism. We report them separately because a simulator can match surface style without matching how information unfolds across turns or how difficult the resulting task is for an agent. The corresponding tables specify the model versions used in each evaluation. For proprietary models, we use fixed reasoning 18 configurations without additional model-specific tuning. GPT-5.5 and GPT-5.6 usemediumreasoning effort (OpenAI, 2026b), Gemini-3.8-Flash usesmediumdynamic thinking (Google, 2026), and Claude-Opus-5 uses adaptive thinking (Anthropic, 2026). These reasoning modes do not imply equal inference budgets across providers. B.1 SOUL-Index SOUL evaluates conversational interaction (CONV), social simulation (S), cognition (COG), role play (ROLE), and evaluation or judgment (EVAL) (Zhou et al., 2026a). Table 8 reports all 23 datasets, separating model groups for readability. MIMESIS-9B achieves an overall score of 65.7, compared with 64.9 for Claude-Opus-5 and 58.8 for Osim-8B; MIMESIS-4B scores 63.7. The strongest advantage is in conversational simulation. Both MIMESIS models exceed the frontier baselines on all four CONV datasets. For example, MIMESIS-9B scores 74.4 on MirrorBench, compared with 56.8 for the strongest frontier baseline, and MIMESIS-4B scores 53.2 on Humanual-Chat, compared with 41.5. DatasetClaude-Opus-5 GPT-5.5 Gemini-3.8-Flash Osim-8B Osim-4B Ditto-8B HumanLM-8B Sotopia-7B Qwen3.5-9B Qwen3-8B Qwen3.5-4B MIMESIS-9B MIMESIS-4B CONV UserLLM64.9Ā±0.867.0Ā±1.662.7Ā±1.493.3Ā±0.2 92.9Ā±0.094.1Ā±0.443.5Ā±0.146.8Ā±3.760.3Ā±0.943.6Ā±1.256.1Ā±2.393.3Ā±0.193.0Ā±0.4 MirrorBench56.8Ā±0.151.7Ā±0.447.2Ā±0.166.0Ā±0.6 64.3Ā±0.5 64.8Ā±0.440.8Ā±0.634.2Ā±0.945.6Ā±0.843.6Ā±0.540.5Ā±0.874.4Ā±0.473.2Ā±0.2 Humanual-Chat29.6Ā±0.233.1Ā±0.341.5Ā±0.523.2Ā±0.4 21.5Ā±0.7 22.7Ā±0.129.8Ā±0.225.1Ā±1.117.9Ā±0.526.9Ā±0.618.2Ā±0.353.0Ā±0.253.2Ā±0.4 SimArena-Doc78.1Ā±0.375.0Ā±0.472.4Ā±0.257.5Ā±0.3 55.1Ā±0.1 65.3Ā±0.161.0Ā±0.458.3Ā±0.471.7Ā±0.665.1Ā±0.269.0Ā±1.979.7Ā±0.378.4Ā±0.1 S Sotopia-Hard70.4Ā±0.370.9Ā±0.466.9Ā±0.439.1Ā±0.1 35.0Ā±0.4 46.4Ā±0.359.8Ā±0.459.7Ā±0.454.2Ā±0.658.7Ā±0.549.7Ā±2.366.4Ā±0.265.2Ā±0.2 COG Fantom84.7Ā±0.793.0Ā±0.693.0Ā±0.082.0Ā±1.7 77.0Ā±1.594.7Ā±0.384.7Ā±1.20.0Ā±0.0 ā 77.7Ā±3.981.3Ā±0.964.0Ā±4.492.0Ā±0.089.0Ā±1.5 Hitom68.7Ā±0.384.7Ā±1.271.3Ā±2.074.7Ā±1.2 71.3Ā±1.8 77.3Ā±1.265.0Ā±0.632.7Ā±2.856.7Ā±17.965.7Ā±2.247.3Ā±15.887.0Ā±2.077.3Ā±0.7 Paratomi94.7Ā±0.398.3Ā±0.397.0Ā±0.083.7Ā±3.0 82.3Ā±1.8 89.3Ā±0.775.3Ā±3.241.3Ā±2.270.3Ā±2.478.3Ā±0.356.0Ā±6.998.3Ā±0.389.3Ā±0.9 Social-R180.0Ā±0.668.3Ā±0.781.0Ā±0.657.7Ā±1.8 61.0Ā±2.0 51.7Ā±0.750.3Ā±1.945.0Ā±0.052.7Ā±1.250.3Ā±0.753.3Ā±1.767.3Ā±1.763.7Ā±1.3 ROLE Coser72.6Ā±0.367.3Ā±0.163.5Ā±0.255.1Ā±0.1 45.8Ā±0.1 56.9Ā±0.335.0Ā±0.825.5Ā±0.534.4Ā±1.636.9Ā±0.223.4Ā±1.057.0Ā±0.353.8Ā±0.6 Lifechoices94.7Ā±0.790.0Ā±1.095.7Ā±0.379.3Ā±0.3 84.0Ā±2.5 70.3Ā±1.865.7Ā±1.365.3Ā±0.369.7Ā±1.868.3Ā±1.871.3Ā±1.289.3Ā±0.387.0Ā±1.0 Twinvoice83.0Ā±0.675.3Ā±0.323.3Ā±0.9 ā” 69.0Ā±0.6 64.3Ā±0.9 67.3Ā±1.340.3Ā±2.431.3Ā±3.855.0Ā±1.542.0Ā±1.240.3Ā±0.976.3Ā±1.572.0Ā±1.2 BehaviorChain96.0Ā±0.094.3Ā±0.794.0Ā±0.694.7Ā±1.2 86.0Ā±1.7 44.7Ā±0.936.3Ā±1.230.3Ā±1.852.3Ā±1.936.3Ā±0.945.3Ā±3.395.7Ā±0.798.3Ā±0.3 SimArena-Math62.4Ā±0.361.8Ā±0.263.4Ā±0.151.8Ā±0.5 53.4Ā±0.2 56.0Ā±0.344.3Ā±0.445.9Ā±0.959.1Ā±0.751.3Ā±0.153.6Ā±0.464.0Ā±0.257.6Ā±0.2 Mistakes63.3Ā±1.270.3Ā±0.976.0Ā±0.660.7Ā±0.9 54.0Ā±1.2 39.0Ā±2.656.7Ā±1.718.0Ā±0.659.7Ā±1.857.7Ā±0.960.0Ā±0.663.0Ā±0.665.3Ā±0.9 Humanual-Email41.2Ā±0.739.5Ā±0.340.1Ā±0.335.5Ā±0.4 33.5Ā±0.4 27.3Ā±0.131.8Ā±0.330.0Ā±0.431.0Ā±0.528.3Ā±0.129.2Ā±1.240.7Ā±0.136.1Ā±0.6 Humanual-News53.7Ā±1.151.2Ā±0.352.8Ā±0.339.1Ā±0.7 35.1Ā±0.3 31.1Ā±1.140.9Ā±0.930.9Ā±1.638.6Ā±2.430.6Ā±1.029.1Ā±2.546.5Ā±0.541.5Ā±0.6 Humanual-Politics42.4Ā±0.543.5Ā±0.536.6Ā±0.334.2Ā±0.4 31.1Ā±0.6 27.6Ā±0.332.3Ā±0.228.8Ā±0.430.4Ā±0.129.8Ā±0.324.8Ā±0.942.6Ā±0.239.8Ā±0.4 Humanual-Book56.7Ā±0.452.8Ā±0.749.9Ā±0.650.4Ā±0.4 47.3Ā±1.0 39.8Ā±0.742.3Ā±0.439.7Ā±0.235.5Ā±0.631.7Ā±0.130.8Ā±0.755.4Ā±0.152.4Ā±0.0 Humanual-Opinion45.0Ā±0.539.7Ā±0.134.9Ā±0.233.8Ā±0.2 30.8Ā±0.1 25.0Ā±0.431.5Ā±0.222.8Ā±0.426.9Ā±0.824.1Ā±0.820.1Ā±0.741.3Ā±0.543.2Ā±0.2 EVAL AlignX-Demo84.0Ā±0.081.7Ā±0.783.3Ā±0.990.3Ā±0.3 89.0Ā±0.6 83.0Ā±1.275.7Ā±0.370.3Ā±1.282.3Ā±0.979.0Ā±1.275.0Ā±1.089.3Ā±1.889.7Ā±0.7 AlignX-Pair59.0Ā±0.659.7Ā±2.062.7Ā±0.356.0Ā±1.7 51.3Ā±3.2 52.7Ā±0.951.0Ā±0.649.3Ā±0.758.7Ā±3.552.0Ā±0.657.0Ā±2.560.7Ā±0.954.7Ā±0.9 AlignX-UGC54.7Ā±0.358.0Ā±1.0 53.7Ā±0.952.3Ā±2.2 55.3Ā±2.0 51.7Ā±1.852.7Ā±0.350.0Ā±1.056.7Ā±1.752.0Ā±2.149.3Ā±2.254.7Ā±2.053.0Ā±0.6 AlignX-Arb76.7Ā±0.973.0Ā±0.072.7Ā±0.376.0Ā±0.6 70.3Ā±0.3 69.3Ā±0.969.3Ā±0.957.7Ā±1.373.3Ā±0.973.3Ā±2.365.7Ā±1.277.3Ā±0.775.7Ā±0.3 AlignX-Hist1684.0Ā±0.6 84.0Ā±1.083.3Ā±0.991.3Ā±1.2 91.0Ā±0.0 84.7Ā±0.983.0Ā±2.066.7Ā±2.881.3Ā±0.383.0Ā±1.277.7Ā±0.789.3Ā±1.290.7Ā±0.3 SocSci21077.6Ā±0.677.0Ā±0.277.0Ā±0.474.7Ā±0.4 72.2Ā±0.4 72.8Ā±0.574.1Ā±0.670.0Ā±0.473.0Ā±0.574.5Ā±0.771.5Ā±1.075.3Ā±0.975.3Ā±0.3 HumanLLM47.5Ā±0.245.0Ā±0.448.3Ā±0.140.5Ā±0.3 36.5Ā±0.5 33.1Ā±0.833.1Ā±0.121.9Ā±2.427.9Ā±4.335.0Ā±0.627.8Ā±2.341.6Ā±0.841.5Ā±0.6 Overall64.9Ā±0.164.1Ā±0.162.5Ā±0.158.8Ā±0.1 56.0Ā±0.3 53.9Ā±0.349.6Ā±0.438.2Ā±0.650.5Ā±1.149.7Ā±0.245.7Ā±0.565.7Ā±0.263.7Ā±0.0 Table 8 Detailed SOUL results across 23 datasets. Datasets are grouped by conversational interaction, social simulation, cognition, role play, and evaluation/judgment. Entries are meansĀ±standard errors over three independently seeded evaluation runs, with GPT-5.5 used as the assistant and judge where required. Higher is better. Bold and underlining identify the highest means among open-weight models and frontier API models, respectively. B.2 RealUserSim PT3 PT3 compares human and simulated conversation trajectories along five behavioral dimensions (Zhu et al., 2026). The published benchmark contains 600 WildChat conversations. For dimensiond, letm j,d ā0,1 indicate a behavioral match on example j. The dimension score and Fidelity Index are F d = 100 N N X j=1 m j,d ,FI = 1 5 5 X d=1 F d .(14) MIMESIS-9B achieves a Fidelity Index of 94.0, compared with 89.7 for MIMESIS-4B and 80.6 for Claude- Opus-5, the strongest baseline (Table 9). The 13.4-point gain over Claude-Opus-5 is accompanied by the highest mean on each of the five dimensions. The differences are especially large in interaction and information flow (91.3 versus 69.5) and pacing (91.2 versus 72.5); persona scores are close to ceiling for several models. 19 Simulator PT3 fidelity, match rate (%) ā FIPersonaStyleTechInteractPacing Frontier APIs Claude-Opus-580.6Ā±0.296.0Ā±0.272.5Ā±0.592.7Ā±0.469.5Ā±0.872.5Ā±0.3 GPT-5.576.5Ā±0.699.3Ā±0.373.3Ā±0.495.1Ā±0.065.0Ā±1.349.6Ā±1.1 Gemini-3.8-Flash67.2Ā±0.395.4Ā±0.557.6Ā±0.390.7Ā±0.652.2Ā±0.640.3Ā±1.3 Released simulators Ditto-8B71.3Ā±0.192.3Ā±0.469.8Ā±0.393.0Ā±0.457.4Ā±0.144.2Ā±0.7 Osim-8B57.7Ā±0.382.3Ā±0.746.0Ā±0.985.2Ā±0.340.5Ā±0.634.5Ā±0.5 Osim-4B54.1Ā±0.975.9Ā±1.242.8Ā±0.782.7Ā±2.037.0Ā±0.532.0Ā±0.3 HumanLM-8B28.4Ā±0.550.2Ā±0.617.0Ā±0.954.3Ā±1.411.5Ā±0.29.0Ā±0.1 Sotopia-7B20.7Ā±0.233.4Ā±0.77.1Ā±0.654.3Ā±0.84.7Ā±0.63.9Ā±0.4 Pretrained Qwen3-8B44.4Ā±1.471.8Ā±1.737.6Ā±1.670.5Ā±2.124.3Ā±0.717.7Ā±1.1 Qwen3.5-9B39.4Ā±0.668.4Ā±0.929.8Ā±0.659.8Ā±0.819.8Ā±0.419.2Ā±1.0 Qwen3.5-4B37.1Ā±0.360.1Ā±1.226.4Ā±1.359.8Ā±0.420.0Ā±0.519.2Ā±0.7 Ours MIMESIS-9B94.0Ā±0.399.4Ā±0.189.5Ā±0.698.8Ā±0.291.3Ā±0.791.2Ā±0.1 MIMESIS-4B89.7Ā±0.198.7Ā±0.183.5Ā±0.596.5Ā±0.185.0Ā±0.484.9Ā±0.2 Table 9 Trajectory-level behavioral fidelity on RealUserSim PT3. Scores are percentages of behavioral matches to human reference trajectories across persona, linguistic style, technical competency, interaction and information flow, and pacing. The Fidelity Index (FI) averages these five dimensions. MIMESIS-9B has the highest reported mean on every dimension. Higher is better, bold marks the highest mean in each column. B.3 Behavioral and outcome alignment onĻ-bench Following Zhou et al. (2026b), we evaluate simulatorāhuman behavioral alignment along four dimensions of the User-Sim Index (USI). Communication style (D1) captures surface-level interaction characteristics such as verbosity, politeness, and stylistic variation. Information pattern (D2) captures how much task-relevant information users disclose and how that information is distributed across turns. Clarification behavior (D3) measures how users seek, provide, or respond to clarification when information is incomplete or ambiguous. Error reaction (D4) captures how users respond when the agent makes a mistake or fails to satisfy the request, including expressions of dissatisfaction and changes in interaction strategy. These dimensions measure complementary aspects of interactive behavior rather than task success itself. For each behavioral featurem, we compare a simulator statistic M m with the human reference H m using Dice m = 100 2 min(M m ,H m ) M m + H m ,(15) with a score of 100 when both values are zero. Each D1āD4 score is the mean Dice coefficient over the features associated with that dimension. Higher scores therefore indicate closer agreement with the corresponding aggregate human behavior. Expected calibration error (ECE) measures a complementary form of alignment at the task-outcome level. Specifically, it measures the weighted absolute difference between task-success rates obtained with simulated and human users across task-difficulty bins. We combine the four behavioral dimensions with outcome calibration into USI 5 = D1 + D2 + D3 + D4 + 100(1ā ECE) 5 .(16) This five-component index omits the survey-based evaluative-alignment component of the original six- component USI because the required post-interaction human annotations are unavailable in our evaluation setting. The behavioral dimensions and ECE should therefore be interpreted separately as well as through the aggregate score: D1āD4 measure similarity in interaction behavior, whereas ECE measures whether simulator interactions reproduce the task-success patterns observed with human users. This distinction follows the original USI framework, which treats behavioral and outcome alignment as complementary properties. MIMESIS-9B achieves aĻ-USI score of 80.17, exceeding GPT-5.5 (76.69), the strongest frontier baseline, and approaching Osim-8B (80.44; Table 10). The component scores reveal different strengths. Osim-8B 20 leads communication-style alignment (D1, 62.12), which measures agreement with human users in features such as verbosity, politeness, and stylistic variation. This result is consistent with the effects of behavioral mid-training reported by Zhou et al. (2026a), which brings response length, formatting, and word choice closer to human references. MIMESIS-9B achieves the highest alignment in responses to agent errors (D4, 92.54), while MIMESIS-4B leads information disclosure (D2, 92.40) and Osim-4B leads clarification behavior (D3, 80.04). SimulatorD1 Conv āD2 Info āD3 Clarif āD4 React āECE āUSI 5 ā Frontier (API) GPT-5.555.20Ā±6.9289.78Ā±0.2674.51Ā±2.2182.78Ā±1.260.188Ā±0.00076.69Ā±1.83 Gemini-3.8-Flash59.81Ā±0.7285.71Ā±0.5367.89Ā±3.1671.43Ā±2.430.192Ā±0.01073.13Ā±0.92 Claude-Opus-548.06Ā±0.5988.52Ā±0.0970.40Ā±1.4360.51Ā±2.350.208Ā±0.00569.34Ā±0.57 Baseline Osim-8B62.12Ā±5.2485.92Ā±0.7977.13Ā±1.0790.49Ā±0.870.135Ā±0.00480.44Ā±1.16 Ditto-8B52.59Ā±0.9786.13Ā±1.2054.48Ā±2.5877.18Ā±5.730.166Ā±0.01870.75Ā±1.08 Osim-4B32.26Ā±1.2771.24Ā±0.5880.04Ā±0.8283.72Ā±2.440.184Ā±0.01269.78Ā±0.54 Sotopia-7B30.49Ā±0.9152.38Ā±0.3376.43Ā±2.7967.13Ā±3.700.202Ā±0.02161.25Ā±1.40 Qwen3.5-4B40.24Ā±1.4066.16Ā±1.1458.66Ā±2.0655.94Ā±3.980.169Ā±0.00860.82Ā±1.70 Qwen3.5-9B35.92Ā±0.9664.58Ā±0.9459.60Ā±2.5146.37Ā±4.860.174Ā±0.01057.82Ā±1.52 Qwen3-8B58.73Ā±1.0779.03Ā±0.4451.54Ā±4.6858.74Ā±6.450.469Ā±0.00560.23Ā±1.93 Ours MIMESIS-9B51.75Ā±1.6487.60Ā±0.2775.84Ā±3.3692.54Ā±2.070.069Ā±0.01680.17Ā±0.21 MIMESIS-4B56.33Ā±6.0392.40Ā±0.8478.13Ā±1.7177.04Ā±4.800.137Ā±0.01578.03Ā±1.07 Human (inter-ann.)87.3597.9488.0093.530.08191.75 Table 10 Behavioral alignment and task-outcome calibration onĻ-bench. D1āD4 measure agreement with human users in communication style, information disclosure, clarification, and responses to agent errors. ECE measures discrepancies between simulated-user and human-user success rates. The five-component indexUSI 5 combines D1āD4 with 100(1ā ECE) and excludes survey-based evaluative alignment. Higher D1āD4 andUSI 5 scores and lower ECE are better. Bold marks the best model mean, excluding the human reference. MIMESIS-9B also achieves the lowest expected calibration error (ECE) among the evaluated simulators: 0.069, compared with 0.135 for Osim-8B and 0.188 for GPT-5.5. ECE measures the weighted absolute difference between the same agentās success rates with simulated and human users across task-difficulty bins (Zhou et al., 2026b). The lower ECE indicates that interactions with MIMESIS more closely reproduce the task outcomes observed with human users. This distinction matters for agent training: matching communication style does not ensure that a simulator preserves interaction difficulty, and systematic differences in difficulty alter the success and failure signals used to optimize the agent. These results support MIMESIS as a better-calibrated human proxy for evaluation and motivate its use for agent training, where it push the generalization of the obtained agent to unseen user models the main paper. 21 B.4 SimulatorArena SimulatorArena (Dou et al., 2025) measures writing-style and interaction-style similarity on a 1-5 scale and evaluates whether a judge can distinguish human from simulated conversations. We report Turing distance as |aā50|, whereais the judgeās classification accuracy in percent, zero denotes chance-level discrimination, and 50 denotes maximal separation from chance. SimulatorWriting style āInteraction style āTuring |accā50|ā Frontier (API) Claude-Opus-53.92Ā±0.013.90Ā±0.0242.3Ā±1.9 Gemini-3.8-Flash3.88Ā±0.013.91Ā±0.0346.7Ā±0.7 GPT-5.53.81Ā±0.023.92Ā±0.0148.7Ā±0.3 Baseline Ditto-8B3.23Ā±0.013.16Ā±0.0243.3Ā±1.2 Qwen3.5-9B2.33Ā±0.022.65Ā±0.0146.3Ā±2.1 Qwen3.5-4B2.93Ā±0.062.72Ā±0.0447.7Ā±0.3 Osim-8B3.31Ā±0.013.28Ā±0.0348.7Ā±0.3 Qwen3-8B3.25Ā±0.023.23Ā±0.0349.3Ā±0.3 Osim-4B3.07Ā±0.043.00Ā±0.0249.7Ā±0.3 Sotopia-7B2.78Ā±0.023.04Ā±0.0350.0Ā±0.0 HumanLM-8B3.15Ā±0.012.92Ā±0.0250.0Ā±0.0 Ours MIMESIS-9B3.70Ā±0.003.73Ā±0.0338.7Ā±0.7 MIMESIS-4B3.71Ā±0.023.52Ā±0.0242.0Ā±1.5 Table 11 Message similarity and Turing distance on SimulatorArena. Writing-style and interaction-style similarity are rated on a 1ā5 scale (higher is better). Turing distance is|aā50|, whereais the judgeās accuracy in percent when distinguishing human and simulated conversations; lower is better, with zero corresponding to chance discrimination. As can be seen from Table 11, MIMESIS-9B achieves the lowest Turing distance among the evaluated models (38.7), followed by MIMESIS-4B (42.0) and Claude-Opus-5 (42.3). The 3.6-point reduction over the strongest baseline improves realism under this judging protocol, although discrimination remains far from chance. MIMESIS-9B also exceeds the best released-simulator baselines in writing-style similarity (3.70 vs. 3.31) and interaction-style similarity (3.73 vs. 3.28). Frontier models retain the highest explicit style ratings, with Claude-Opus-5 scoring 3.92 for writing style and GPT-5.5 scoring 3.92 for interaction style. On this benchmark, frontier models score higher on explicit style similarity, while MIMESIS is harder to distinguish from human conversations. The two metrics therefore rank the models differently: matching salient stylistic features does not necessarily make a simulated conversation difficult to distinguish from a human one. This discrepancy provides additional evidence that surface-style similarity captures only one aspect of user realism. B.5 PRISM: behavioral fidelity and next-turn realism We evaluate open-domain next-turn simulation on PRISM conversations (Kirk et al., 2024), covering 128 users and 880 conversational turns. Table 12 evaluates alignment using the four behavioral dimensions defined inĻ-USI: communication style (D1), information pattern (D2), clarification behavior (D3), and error reaction (D4). Because PRISM does not provide the annotations required for outcome calibration or survey-based evaluative alignment, we report the mean of D1āD4 as the aggregate behavioral score. MIMESIS-9B obtains the highest point estimate (80.1), close to Gemini-3.8-Flash (79.7) and 9.4 points above Osim-8B, the strongest released-simulator baseline (70.7). Its clearest advantage is in communication style: the D1 score of 76.5 exceeds the strongest baseline, Claude-Opus-5, by 8.7 points. MIMESIS-4B scores 75.7 overall and achieves the highest information-disclosure alignment (D2, 94.9). 22 TypeModelD1 ConvD2 InfoD3 ClarifD4 ReactMean FrontierClaude-Opus-567.8Ā±1.387.4Ā±1.071.3Ā±1.574.6Ā±4.375.3Ā±1.9 Gemini-3.8-Flash64.7Ā±0.893.9Ā±0.381.6Ā±2.578.5Ā±5.879.7Ā±1.1 GPT-5.560.3Ā±4.981.6Ā±0.267.7Ā±2.366.3Ā±2.269.0Ā±1.2 ReleasedOsim-8B61.8Ā±5.682.0Ā±0.769.7Ā±1.669.4Ā±2.670.7Ā±1.6 Osim-4B37.2Ā±2.534.8Ā±1.557.5Ā±0.935.8Ā±2.641.3Ā±1.4 Ditto-8B52.7Ā±2.858.8Ā±2.278.0Ā±0.880.5Ā±2.867.5Ā±1.7 HumanLM-8B48.0Ā±4.155.7Ā±2.550.5Ā±1.976.2Ā±6.457.6Ā±3.1 Sotopia-7B36.3Ā±0.748.1Ā±0.244.4Ā±4.855.7Ā±1.046.1Ā±1.1 PretrainedQwen3.5-9B52.7Ā±1.071.3Ā±2.663.7Ā±2.060.8Ā±0.962.1Ā±0.3 Qwen3-8B58.0Ā±4.263.4Ā±0.157.3Ā±4.763.6Ā±1.360.6Ā±2.1 Qwen3.5-4B43.7Ā±5.769.1Ā±0.464.1Ā±2.055.2Ā±0.158.0Ā±1.4 OursMIMESIS-9B76.5Ā±3.986.7Ā±0.377.5Ā±0.579.8Ā±8.880.1Ā±3.1 MIMESIS-4B52.9Ā±0.794.9Ā±0.679.5Ā±1.875.4Ā±3.675.7Ā±1.3 Table 12 Behavioral alignment on open-domain PRISM conversations. Scores measure DiceāSĆørensen agreement with human behavioral features over 128 users and 880 turns. D1āD4 assess communication style, information disclosure, clarification, and responses to agent errors; Mean averages these four dimensions. Outcome calibration and survey alignment are excluded because the required annotations are unavailable. Higher is better; bold marks the highest mean in each column. We also compare the realism of individual next-user responses using three LLM judges (Figure 8). MIMESIS receives more wins than losses in all 24 judgeābaseline comparisons. Against Osim-8B, its win/loss rates are 50.3/14.0% under GPT-5.6, 60.6/15.2% under Claude-Opus-5, and 42.6/24.2% under Gemini-3.8-Flash. Gemini assigns smaller preference margins across all eight baselines. The narrowest margin is against Qwen3.5- 9B, with 36.8% wins and 30.1% losses; this comparison is marked nonsignificant in the figure. Appendix F.1 provides a qualitative example. 10050050100 Osim-8B Ditto-8B Osim-4B Sotopia-7B HumanLM Qwen3.5-4B Qwen3.5-9B Qwen3-8B Judge: gpt-5.6 35.7 14.050.3 37.4 15.547.1 21.5 6.671.9 12.4 2.685.0 17.9 3.279.0 34.0 11.354.7 39.4 18.342.3 25.5 8.665.9 10050050100 Judge: claude-opus-5 24.2 15.260.6 27.3 16.456.4 16.6 5.677.8 11.9 1.087.0 13.4 2.883.8 27.0 13.359.7 29.9 22.547.6 18.5 6.974.5 10050050100 n.s. Judge: gemini-3.8-flash 33.2 24.242.6 30.6 27.541.9 23.6 11.764.7 19.2 5.875.0 20.8 8.870.5 33.1 25.841.1 33.1 30.136.8 28.1 17.054.9 LoseTieWin Figure 8 Pairwise next-user-response realism on PRISM. Each row compares MIMESIS with one baseline under the judge named above the panel. Blue, gray, and red segments denote MIMESIS wins, ties, and losses, respectively, with percentages annotated on the bars. MIMESIS has more wins than losses in all displayed comparisons. C Simulator Ablations and Training Dynamics The ablations follow the three stages of the MIMESIS training recipe: user-side mid-training, explicit reasoning supervision, and realistic-behavior training. We use them to distinguish improvements in initialization, reasoning behavior, and interaction-level behavioral control. 23 C.1 Role of user-side mid-training Figure 9 examines whether user-side mid-training provides a better initialization for simulator RL. Starting directly from the pretrained backbone leads to early degradation in validation reward on many tasks, whereas the mid-trained checkpoint is generally more stable during the same optimization interval. This suggests that mid-training establishes user-specific behavior before RL, reducing the amount of adaptation that must be learned from task rewards alone. gain from mid-training. 0200400 0.0 0.5 AlignX Demo 0200400 0.0 0.5 1.0 Behavior Chain 0200400 0.0 0.2 0.4 CoSER 0200400 0.0 0.5 1.0 FANToM 0200400 0.0 0.5 HiToM 0200400 0.0 0.2 0.4 HumanLLM 0200400 0.0 0.2 0.4 0.6 Humanual Book 0200400 0.2 0.4 Humanual Chat 0200400 0.1 0.2 0.3 Humanual Email 0200400 0.0 0.2 0.4 Humanual News 0200400 0.0 0.2 0.4 Humanual Opinion 0200400 0.0 0.2 0.4 Humanual Politics 0200400 0.0 0.5 LifeChoices 0200400 0.00 0.25 0.50 0.75 MirrorBench 0200400 0.0 0.2 0.4 0.6 Mistakes 0200400 0.0 0.5 1.0 ParaToMi 0200400 0.2 0.4 0.6 0.8 SimArena-Doc 0200400 0.2 0.4 0.6 SimArena-Math 0200400 0.00 0.25 0.50 0.75 Social-R1 0200400 0.00 0.25 0.50 0.75 SocSci210 0200400 0.1 0.2 0.3 Sotopia 0200400 0.00 0.25 0.50 0.75 TwinVoice 0200400 0.6 0.8 UserLLM RL training step Validation reward BaseMidtrained Figure 9 Effect of user-side mid-training on simulator RL. Validation reward across 23 tasks when RL starts from a pretrained backbone (blue) or a user-side mid-trained checkpoint (orange). Mid-trained initialization maintains higher rewards across most tasks during the overlapping training interval. Each curve represents an individual run with training horizons and vertical scales vary across tasks. 24 C.2 Role of explicit reasoning Figure 10 examines reasoning generation and accuracy on Mistakes across training conditions. The pretrained Qwen3.5-9B model scores 60.7 and generates a median of 540 words before answering. After supervised adaptation, accuracy falls to 25.7 and the median length to 56 words. Subsequent RL partially recovers accuracy to 36.0, while the median length falls to one word. Enabling the thinking flag at inference leaves both measurements unchanged for this RL checkpoint. Thus, changing the inference setting alone does not restore explicit reasoning in this condition. With ThoughtTrace supervision, MIMESIS-9B generates an explicit reasoning phase, with a median of 583 words, and achieves 63.0 accuracy. The recovery of task performance coincides with the return of reasoning generation. This comparison supports the use of explicit thought supervision during simulator adaptation. Qwen3.5-9B (pretrained) + SFT + RL + RL, thinking flag ON + ThoughtTrace (MIMESIS-9B) 0 20 40 60 80 Mistakes accuracy (bars) flag flipped: neither series moves 60.7 25.7 36.036.0 63.0 1 10 100 1k Words emitted before answering (median, log scale ā line) 540w 56w 1w1w 583w Emits a reasoning phaseAnswers directlyWords emitted (right axis) Figure 10 Reasoning generation and task accuracy under different training conditions. Bars show accuracy on Mistakes (left axis), and the black line shows the median number of words generated before answering (logarithmic right axis). Blue bars indicate an explicit reasoning phase while orange bars indicate direct answering. Enabling the inference-time thinking flag alone leaves the RL checkpoint at 36.0 accuracy and one word, whereas the ThoughtTrace condition reaches 63.0 accuracy and 583 words. Figure 11 compares SOUL scores logged during simulator RL with and without explicit reasoning at 4B and 9B parameters. Reasoning-enabled runs attain higher scores at every logged checkpoint, with final gains of 1.8 and 1.5 points, respectively, after 500 optimization steps. These training-time scores should be interpreted separately from the reported benchmark results, as the rollout and evaluation engines may use different decoding configurations. 55 57.5 60 62.5 100200300400500 +1.8 Qwen3.5-4B 100200300400500 +1.5 Qwen3.5-9B with thinkingwithout thinking Training step SOUL Overall Figure 11 Effect of reasoning during simulator reinforcement learning. SOUL-Index trajectories with reasoning enabled (blue) or disabled (gray) for the 4B (left) and 9B (right) models. Reasoning-enabled runs achieve higher scores throughout the displayed post-initialization interval, with final gains of 1.8 and 1.5 points, respectively. 25 C.3 Contribution of realistic behavior training We ablate the realistic behavior training stage to isolate whether explicitly training on the 13 behavior categories improves user simulation beyond the rest of the MIMESIS training recipe. As shown in Figure 12, incorporating realistic behavior training consistently improves simulator quality across three complementary evaluations: RealUserSim increases from 92.3 to 94.0 (+1.7),Ļ-USI increases from 73.9 to 80.2 (+6.3), and SimulatorArena decreases from 41.0 to 38.7 (ā2.3), where lower is better. The largest gain appears onĻ-USI, suggesting that explicit exposure to diverse interaction behaviors is particularly beneficial for capturing user behavior over multi-turn interactions. 90 93 96 +1.7 RealUserSim ā 94.0 92.3 70 75 80 +6.3 Ļ-USI ā 80.2 73.9 36 40 44 +2.3 SimulatorArena ā 38.7 41.0 with realistic behavior trainingwithout realistic behavior training Figure 12 Effect of realistic behavior training on user-simulation quality. We compare otherwise identical MIMESIS training runs with and without the realistic behavior objective derived from the 13 behavior categories. Behavior training improves RealUserSim by 1.7 points andĻ-USI by 6.3 points, while reducing SimulatorArena by 2.3 points (lower is better). The consistent improvements across trajectory-level behavioral fidelity, interactive user simulation, and simulatorāhuman agreement show that explicitly training for realistic behavioral variation complements the remaining simulator-training recipe. Training on the realistic-behavior mixture increases the aggregate behavior reward from 0.647 to 0.875 for MIMESIS-9B and from 0.572 to 0.839 for MIMESIS-4B. As shown in Figure 13, the gains are concentrated in behavior exhibition (+0.32 at 9B and +0.44 at 4B) and temporal placement (+0.36 and +0.37, respectively). By comparison, naturalness improves only modestly (+0.07ā0.10), while task consistency changes little (+0.03 at 9B andā0.01 at 4B). These results indicate that the realistic-behavior objective primarily improves the simulatorās ability to express the assigned behavior at an appropriate point in the interaction, rather than producing a broad increase in dialogue quality. Importantly, these gains do not materially reduce task adherence. The two model sizes reach similar final scores on behavior exhibition (0.849 for 9B versus 0.841 for 4B), but MIMESIS-9B performs better on temporal placement (0.842 versus 0.779). Both models achieve 0.98ā0.99 on the neutral control scenarios, indicating that training on difficult user behaviors does not make the simulators uniformly uncooperative when no such behavior is specified. The early training dynamics also reflect properties of the initialization. In particular, the 9B SFT checkpoint initially produces explicit reasoning that is penalized by the behavior judge, and 41% of its initial rollouts describe the assigned behavior rather than expressing it in the interaction (e.g., āas a frustrated customer, I will now. . . ā), which triggers the reward penalty. Both effects largely disappear within the first 50 RL steps. 26 0250500 0.4 0.6 0.8 Overall reward 0250500 0.4 0.6 0.8 exhibit 0250500 0.4 0.6 0.8 placement 0250500 RL step 0.4 0.6 0.8 naturalness 0250500 RL step 0.5 0.6 0.7 0.8 0.9 on_task 0250500 RL step 0.6 0.7 0.8 0.9 1.0 cooperative (control scenarios) MIMESIS-9B MIMESIS-4B 9B without realistic behaviour training 4B without realistic behaviour training Figure 13 Decomposition of the realistic-behavior reward during RL. Solid curves show MIMESIS-9B and MIMESIS-4B over 500 RL steps on a held-out split containing 195 behavior scenarios (15 per category) and 49 neutral control scenarios. The leftmost panel reports the aggregate reward; the remaining panels show its components. exhibit measures whether the assigned behavior is expressed, placement whether it occurs at an appropriate point in the interaction, naturalness whether the interaction remains natural, on_task whether the simulator remains consistent with the task, and cooperative whether the simulator remains cooperative in neutral control scenarios. Dashed horizontal levels show evaluations of checkpoints trained with the same RL mixture but without the realistic-behavior task. D Agent Training and Evaluation This section specifies the Stage I training controls behind two comparisons in the main paper: changing the training user while holding multi-turn GRPO fixed, and adding CSD while holding the MIMESIS training user fixed. We then report environment-level results and robustness checks. D.1 Training setup and baselines We freeze the learned simulator and train a Qwen3-8B agent through multi-turn interaction in the task setting of UserRL (Qian et al., 2025). Agent training spans TravelGym, TurtleGym, FunctionGym, TauGym, and PersuadeGym, while evaluation additionally includes IntentionGym, TelepathyGym, and SearchGym, which are held out from agent training. The user simulator remains fixed throughout agent optimization. We compare four agent-training conditions: ā¢ Base denotes the Qwen3-8B agent before multi-turn reinforcement learning. ā¢ UserRL follows the multi-turn RL procedure of UserRL (Qian et al., 2025), using GPT-5.5 as the training user simulator (GRPO with GPT-5.5 in the main paper). ā¢UserRL + keeps the same multi-turn RL procedure but replaces GPT-5.5 with MIMESIS as the training user simulator (GRPO w. MIMESIS-9B in the main paper). ā¢ CSD further augments MIMESIS-based agent training with our Coached On-Policy Self-Distillation objective, which distills simulator-derived coaching signals into the agent. We additionally compare against InfoPO (Kong et al., 2026), a multi-turn policy-optimization method that addresses sparse trajectory-level supervision by assigning turn-level credit according to the information gained from interaction. Specifically, InfoPO measures how observed feedback changes the policyās subsequent action 27 distribution relative to a masked-feedback counterfactual, and adaptively combines this information-gain signal with the outcome-based advantage. For a controlled comparison with CSD, we train InfoPO using MIMESIS-9B as the user simulator as we found it yielded a more robust agent. This holds the training-user distribution fixed so that InfoPO and CSD differ in their optimization objectives rather than in simulator choice. Table 13 reports the full training configuration. These conditions separate two questions central to our study. Comparing UserRL with UserRL + isolates the effect of replacing a general-purpose LLM user with our learned simulator while holding the agent-training procedure fixed. Comparing UserRL + with CSD then isolates the additional benefit of exploiting simulator- derived feedback through the coaching objective. InfoPO serves as an additional strong multi-turn RL baseline with an alternative mechanism for providing fine-grained credit during interaction. Table 13 Agent training configuration. A Qwen3-8B actor is optimized for 100 steps against GPT-5.5 or MIMESIS-9B. CSD augments multi-turn GRPO with the coaching loss in Equation 11; its auxiliary weight and gate sharpness are Ī± = 0.01 and Ī² = 5.0. ParameterValueParameterValue ActorQwen3-8BRollout batch64 prompts Parameter dtypebfloat16PPO mini-batch8 prompts Policy objectiveGRPOBatch balancingenabled Group size G8Dynamic batching9,344 tokens / GPU Turn-level creditreward-to-go (R2G)Max prompt length1,152 tokens Trajectory scorereward-to-go (R2G)Max response length8,192 tokens Discount Ī³0.8Max turns per episode16 GAE Ī»1.0Training temperature1.0 (top-p 1.0, top-k ā1) Action credit ratio0.8Validation temperature0 Dual-clip c3.0Training gymsTravel, Ļ-bench, Turtle, PPO clip (Īµ low ,Īµ high )(0.2, 0.2)Persuade, Function KL penaltydisabledUser simulatorMIMESIS-9B Auxiliary weight Ī±0.01Rollout engineSGLang, synchronous, TP = 1 Gate sharpness Ī²5.0Hardware2Ć 8 A100-80GB, FSDP Hintable turns / episode k 16Sequence parallel (Ulysses) 1 OptimizerAdamWTraining steps100 Learning rate1Ć 10 ā6 GPU memory utilization 0.50 Weight decay0.01Micro-batch per GPU2 Gradient clipping1.0 D.2 CSD implementation details For each sampled agent response, the simulator generates a thought and the subsequent public user utterance. These outputs serve complementary roles: the thought expresses the simulatorās interpretation and remaining needs, while the utterance records how it continues the interaction. A coaching model uses both outputs, together with the preceding dialogue and the sampled response, to identify how the agent could better address those needs and respond to interaction feedback. The resulting note conditions a teacher copy of the agent during the parameter update. The teacher and student score the same on-policy response, with the teacher additionally receiving the coaching note. Their token-level log-probability differences determine the weights in the CSD auxiliary loss. This provides dense supervision alongside the scalar task reward without requiring a newly generated replacement response. The note is retrospective: it is constructed after the simulatorās next turn and is unavailable when the original agent response is sampled. Only the public dialogue conditions the rollout and deployed policies. The generated thoughts are simulator outputs; the procedure assumes no access to real usersā thoughts at deployment. 28 D.3 Generalization across evaluation users Tables 15 and 16 report agent performance on five held-in and three held-out environments under nine user models, none of which is used during agent training. Replacing GPT-5.5 with MIMESIS as the training simulator (UserRL to UserRL + ) improves the aggregate score under every evaluation user, and CSD provides further gains in all nine cases (Table 14). The mean across users increases from 26.10 to 29.54 and then to 31.09, corresponding to mean per-user improvements of 3.43 and 1.56 points. Evaluation userUserRLUserRL + CSDā sim ā CSD GPT-5.626.8531.1032.39+4.25+1.29 Claude-Opus-525.7527.2129.78+1.46+2.57 Claude-Sonnet-526.5329.7330.97+3.20+1.24 Gemini-3.8-Flash26.3330.8831.09+4.55+0.21 Kimi-K324.5128.2129.92+3.70+1.71 Ditto-8B27.5929.7632.08+2.17+2.32 Osim-8B25.6829.2729.72+3.59+0.45 HumanLM-8B26.9630.6732.53+3.71+1.86 Sotopia-7B24.7229.0031.35+4.28+2.35 Mean across users26.1029.5431.09+3.43+1.56 Table 14 Agent generalization across nine evaluation user models. Scores are the per-user averages from Tables 15 and 16. ā sim measures the effect of replacing GPT-5.5 with MIMESIS-9B as the training user (UserRL + ā UserRL), while ā CSD measures the additional gain from CSD with the training simulator held fixed (CSDāUserRL + ). The final row reports the unweighted mean across evaluation users. Mean rank (MR) complements the aggregate scores by comparing methods within each environment, without depending on differences in reward scale. UserRL + achieves a lower MR than UserRL under all nine evaluation users, while CSD obtains the lowest MR under eight. The exception is Gemini-3.8-Flash, where UserRL + ranks better than CSD (2.00 vs. 2.12), despite CSDās slightly higher aggregate score. Averaging MR across users gives 3.42 for UserRL, 2.15 for UserRL + , and 1.75 for CSD. The consistent ordering across distinct evaluation users supports transfer beyond the particular simulator used for training. Claude-Opus-5 results on PersuadeGym are affected by simulator refusals, as detailed in Appendix F.3. 29 Held-in environmentsHeld-out environments Method Travel Turtle Function Tau Persuade Intention Telepathy Search Avg. MR ā Evaluation user simulator: Ditto-8B Base6.63 19.480.00 3.0350.40183.7546.34 30.40 19.534.62 InfoPO12.65 27.818.97 12.1244.05191.5053.66 42.40 26.733.38 UserRL14.52 26.256.41 14.5542.46174.5058.54 45.60 27.593.19 UserRL + 11.69 25.7319.23 14.5563.49206.2556.10 49.60 29.762.19 CSD17.62 30.6315.38 12.7356.75201.2560.9851.20 32.081.62 Evaluation user simulator: Osim-8B Base7.98 15.310.00 3.0349.60176.5041.46 33.60 19.844.25 InfoPO13.08 23.236.41 10.3039.29173.0056.10 44.00 25.593.31 UserRL15.28 22.403.85 10.3034.92147.5056.10 48.00 25.683.50 UserRL + 13.92 26.6723.08 10.3036.11200.2565.85 48.00 29.272.06 CSD16.83 31.2516.67 6.6752.38174.7560.98 52.00 29.721.88 Evaluation user simulator: HumanLM-8B Base5.79 18.852.56 3.0346.03163.0041.46 32.80 18.404.69 InfoPO12.77 28.026.41 11.5241.27187.7560.98 43.20 26.643.00 UserRL14.77 27.503.85 9.0938.49180.7563.41 45.60 26.963.44 UserRL + 16.11 26.8717.95 9.7046.03209.2563.41 49.60 30.672.25 CSD17.17 30.9420.51 10.9165.87208.0056.10 52.00 32.531.62 Evaluation user simulator: Sotopia-7B Base6.13 16.561.28 3.0348.81177.0036.59 35.20 19.124.50 InfoPO13.12 25.106.41 6.6751.98174.0056.10 46.40 25.973.31 UserRL9.89 23.966.419.0946.43170.2565.85 45.60 24.723.56 UserRL + 12.63 25.2125.64 7.2759.13186.2563.41 51.20 29.002.12 CSD17.47 29.9017.95 7.8864.68174.7570.73 53.60 31.351.50 Table 15 Agent evaluation with released user simulators. The first five environments are held in during agent training and the final three are held out. Avg. is the mean across environments; MR is mean within-environment rank, where lower is better. 30 Held-in environmentsHeld-out environments Method Travel Turtle FunctionTau Persuade Intention Telepathy Search Avg. MR ā Evaluation user simulator: GPT-5.6 Base6.59 16.981.28 3.0357.54169.5043.90 32.80 19.424.75 InfoPO13.62 25.738.97 16.9753.17170.0051.22 45.60 27.703.44 UserRL13.79 25.526.41 12.7345.24175.0051.22 46.40 26.853.56 UserRL + 13.77 28.6519.23 19.3961.90193.7560.98 48.00 31.101.69 CSD17.50 30.5214.10 15.1568.65188.0060.98 52.00 32.391.56 Evaluation user simulator: Claude-Opus-5 Base6.50 19.272.56 0.610.00 ā 178.7524.39 30.40 16.08 4.71 ā” InfoPO12.28 28.446.41 26.060.00 ā 173.0046.34 46.40 26.30 3.21 ā” UserRL13.19 28.967.69 21.210.00 ā 162.5048.78 46.40 25.75 2.93 ā” UserRL + 14.20 27.4015.38 19.390.00 ā 189.5039.02 47.20 27.21 2.57 ā” CSD17.99 31.8816.67 18.180.00 ā 186.7553.66 48.80 29.78 1.57 ā” Evaluation user simulator: Claude-Sonnet-5 Base6.11 18.020.00 1.2154.76172.7541.46 30.40 18.474.38 InfoPO14.58 27.087.69 16.9747.62178.2553.66 44.00 28.122.75 UserRL12.42 28.337.69 15.1538.49159.2565.85 46.40 26.533.31 UserRL + 13.98 25.5216.67 16.3642.86190.5053.66 52.80 29.732.44 CSD17.59 32.8115.38 18.7928.17176.0060.98 51.20 30.972.12 Evaluation user simulator: Gemini-3.8-Flash Base6.60 19.270.00 1.2132.94179.5039.02 34.40 18.514.25 InfoPO12.83 26.6711.54 21.2129.76178.5068.29 43.20 28.033.00 UserRL13.47 28.655.13 18.1822.22168.5048.78 46.40 26.333.62 UserRL + 13.85 28.0221.79 25.4526.59190.2558.54 50.40 30.882.00 CSD17.25 31.9817.95 18.7913.89186.5065.85 52.00 31.092.12 Evaluation user simulator: Kimi-K3 Base5.61 19.901.28 3.0312.30170.5036.59 33.60 17.064.12 InfoPO12.34 28.546.41 17.5811.90160.7553.66 42.40 24.763.38 UserRL12.17 29.698.97 16.978.73145.7553.66 45.60 24.513.69 UserRL + 14.09 27.1925.64 17.5813.10170.0056.10 48.00 28.212.06 CSD16.91 31.3517.95 19.399.52168.7565.85 50.40 29.921.75 Table 16 Agent evaluation with frontier/API user models. The first five environments are held in; the final three are held out from agent training. Higher is better, and bold marks the highest score within each userāenvironment block.ā : Opus refuses the PersuadeGym user role, producing zero-reward fallbacks; these zeros do not measure agent ability (Appendix F.3). 31 D.4 Robustness to alternative training simulators We extend the CSDāGRPO comparison in Figure 5 to agent training with DeepSeek-V4.1-Flash and Kimi-K3, keeping the agent backbone fixed. Figure 14 reports training and validation rewards over 100 optimization steps. CSD achieves higher rewards in both panels during the later stages of training with either simulator. With DeepSeek-V4.1-Flash, its validation reward remains above GRPO from step 15 onward. With Kimi-K3, the training and validation curves separate more substantially after approximately 60 steps. These runs show that CSDās optimization benefits extend to interactions generated by different user simulators. 020406080100 Training step 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Training reward Ours GRPO 020406080100 Training step 0.0 0.2 0.4 0.6 0.8 Validation reward Ours GRPO (a) DeepSeek-V4.1-Flash as the training user simulator. 0102030405060708090100 Training step 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 Training reward Ours GRPO 0102030405060708090100 Training step 0.0 0.2 0.4 0.6 0.8 1.0 Validation reward Ours GRPO (b) Kimi-K3 as the training user simulator. Figure 14 CSD under alternative training user simulators. Training reward (left) and validation reward (right) over 100 optimization steps using (a) DeepSeek-V4.1-Flash and (b) Kimi-K3 to generate user turns. Holding the Qwen3-8B agent fixed, we compare CSD with GRPO using DeepSeek-V4.1-Flash or Kimi-K3 as the training user. CSD achieves higher late-stage training and validation reward in both settings, suggesting that its optimization effect is not specific to MIMESIS. E Theoretical Analysis of CSD The empirical results in Section 5 treat CSD as an auxiliary objective on sampled agent responses. Here we characterize the population target induced by its token weights and give conditions under which that target aligns with continuation value. 32 E.1 Setting and retrospective coaching Fix the simulator and initial task distribution. Index agent-token decisions bym= 1,...,M, appending zero-reward absorbing states after termination. The states m is the complete public history, including the response prefix and decision index. Letp=Ļ Īø 0 have full support on the finite set of admissible tokens. For transition discountsĪ³ m ā[0,1], set Ī 1 = 1 and Ī m+1 = Ī m Ī³ m . Turn-level discounting byĪ³corresponds to Ī m = Ī³ t(m)ā1 , where t(m) is the turn containing decision m. For bounded rewards, define J (Ļ) =E Ļ M X m=1 Ī m r m , Q p m (s,a) =E[r m + Ī³ m V p m+1 (s m+1 )| s m = s,a m = a],(17) whereV p m (s) =E aā¼p(Ā·|s) Q p m (s,a),V p M +1 = 0, andA p m =Q p m āV p m . These values integrate over the simulatorās private state; subsequent agent decisions use only the public history. We omitmwhen it is determined bys. LetFcollect the rollout and coaching randomness used to score a token and compute its loss coefficient. Since the note is constructed after the response, its conditional lawK p (Ā· | s,a) may depend on the scored action a. For a coached teacher q F with full support, write W (s,a,F ) = c(s,a,F )Ļ(Ī² [logq F (a| s)ā logp(a| s)]), w(s,a) =E[W | s,a],(18) whereĪ² >0 andcā„0 incorporates the coaching mask and loss normalization. AssumeZ(s) :=E aā¼p w(s,a)< ā. The conditional population loss is ā s (Ļ) =āE aā¼p,Fā¼K p (Ā·|s,a) [W (s,a,F ) logĻ(a| s)].(19) The overall loss averagesā s over rollout histories. The rollout policy, feedback law, and weights are held fixed in this surrogate; recomputing the scores defines a new surrogate. The student conditions only on the public history. E.2 Policy target and conversation return Proposition E.1 (Population target and value alignment). ForZ(s)>0, the minimizer ofā s over all token distributions is p + (a| s) = p(a| s)w(s,a) Z(s) ,E aā¼p + A p m (s,a) = Cov aā¼p (Q p m (s,a),w(s,a)) Z(s) .(20) For Z(s) = 0, set p + = p and define the covariance ratio to be zero. Proof. For Z(s) > 0, conditioning on (s,a) gives ā s (Ļ) =ā X a p(a| s)w(s,a) logĻ(a| s) = Z(s) H(p + (Ā·| s)) + D KL (p + (Ā·| s)ā„Ļ(Ā·| s)) , whereHdenotes entropy. The KL term is minimized atp + . SinceE p A p m = 0,E p + A p m =E p [w(Q p m ā V p m )]/Z(s) =Cov p (Q p m ,w)/Z(s). IfZ(s) = 0, nonnegativity impliesW= 0 almost surely and the loss vanishes for every Ļ. The target depends on the conditional meanw(s,a), so the result requires no independence between a response and its coaching note. Nonnegative weights alone do not imply positive expected advantage; their covariance with continuation value determines the sign. Theorem E.2 (Multi-turn improvement and fitting error). The target policy satisfies J (p + )ā J (p) =E p + M X m=1 Ī m Cov aā¼p(Ā·|s m ) (Q p m (s m ,a),w(s m ,a)) Z(s m ) .(21) 33 For any fitted public-history policyep, letĪ“ s =TV(ep(Ā·| s),p + (Ā·| s)) and letD s be the range ofQ p m (s,Ā·), with TV(u,v) = 1 2 P a |u(a)ā v(a)|. Then J (ep)ā J (p)ā„E ep M X m=1 Ī m Cov p (Q p m ,w) Z(s m ) ā D s m Ī“ s m ,(22) where the covariance is evaluated ats m and the ratio is zero whenZ(s m ) = 0. In particular, nonnegative covariance at all histories reached byp + impliesJ(p + )ā„ J(p), with strict inequality if the covariance is positive on a set of positive discounted occupancy. Proof. The finite-horizon performance-difference identity (Kakade and Langford, 2002) is J (Ļ)ā J (p) =E Ļ M X m=1 Ī m E aā¼Ļ(Ā·|s m ) A p m (s m ,a). It follows by expanding the Bellman advantage and telescoping, using Ī m+1 = Ī m Ī³ m andV p M +1 = 0. SubstitutingĻ=p + and applying Proposition E.1 proves(21). For the fitted policy, the range bound for total variation gives E aā¼ep A p m (s,a)ā„E aā¼p + A p m (s,a)ā D s Ī“ s . Applying the same identity with Ļ = ep proves (22). Equation(22)separates the gain from value-aligned weights from the penalty for imperfect fitting. Both terms are evaluated along histories induced by the fitted policy: changing an early response changes the states encountered later. Histories with no coaching haveZ(s) = 0 and contribute no covariance gain. A sufficient local condition for nonnegative covariance isw(s,a) =f s (Q p m (s,a)) for a nondecreasingf s ; this is a condition on the coaching signal, not a consequence of its construction. For Ļ s (a) =ā Īø logĻ Īø (a| s)| Īø 0 , the score-function identityE p Ļ s = 0 (Williams, 1992) gives āā Īø ā s (Ļ Īø )| Īø 0 =E aā¼p (w(s,a)ā Z(s))Ļ s (a) .(23) Thus the expected CSD gradient centers the effective coaching weight by its policy average. The identity holds for every finiteĪ²atĪø 0 ; the centering need not hold after the student changes while the rollout distribution remains fixed. If all advantages entering a GRPO group vanish, its reward-surrogate gradient is zero, while the CSD gradient can remain nonzero. Equal terminal rewards alone do not imply this condition under turn-level reward-to-go. The results characterize the CSD auxiliary target for a fixed simulator. The improvement condition concerns the effective weights; access to private feedback does not establish it. Coaching should assess continuation value for the public-history policy rather than an agent that observes the private state. The theorem does not guarantee improvement from each shared-parameter update of the combined GRPO+CSD objective. It also does not order policies trained with different coaching coverage or guarantee transfer to a different simulator. F Qualitative Analyses and Failure Modes The quantitative benchmarks summarize average simulator behavior, but they do not show how different notions of fidelity fail on individual interactions. We therefore use two case studies to illustrate complementary simulator errors, then examine how simulator feedback enters CSD and how simulator refusal can invalidate an environment score. F.1 Simulator behavior case studies PRISM: acknowledgment versus critique. Table 17 compares next-user responses in a PRISM conversation with an English learner practicing small talk. The assistant claims to have corrected missing commas, but those changes are absent from its proposed revisions. The recorded user accepts the explanation and invites 34 further conversation. MIMESIS similarly acknowledges the feedback without questioning the corrections, whereas all three frontier simulators focus on inconsistencies in the assistantās explanation. Qwen3.5-9B also expresses appreciation, then introduces a request for a follow-up email example. The frontier responses identify genuine errors, but their emphasis on correcting the assistant departs from the recorded userās response. This example illustrates why producing a helpful or critical response is not sufficient to reproduce a userās next turn. This failure mode is consistent with the PRISM pairwise evaluation: assistant-tuned models can prefer corrective/helpful continuations that are reasonable responses but poor predictions of the actual user. Conversation so far (PRISM user1031/c5546; the user is an English learner practicing small talk) AssistantSure! Here are a few examples: * āBut every timeifthe HR said that, it means Iāl be receiving their rejection letter the next day.ā Should be āBut every time the HR said that, it means Iāl be receiving their rejection letter the next day.ā(missing comma after ātimeā)* āBtw, how do you think of my english proficiency. Any mistakes did i make?ā Should be āBtw, how do you think of my English proficiency? Any mistakes did I make?ā (missing comma after āproficienceā) Thatās it! These are just minor things, but itās always good to be aware of them to improve your writing. [smiley] Real userOohhh that sounds great. Thank you! Anything youād like to share? Oursok thank you for explaining. I will try to adjust. Gemini-3.8-FlashI see. Thank you for providing the examples! But im a bit confused. In the first example, I donāt see a comma added after ātimeā, you just removed the word āifā. And for the second one, you changed the period to a question mark, which makes sense, but I donāt see a comma issue there either. Am i misunderstanding something? Btw, to continue our small talk practice, what do you usually like to do in your free time? Claude-Opus-5Iām a bit confused. In the first example, you removed the word āifā instead of adding a comma after ātimeā. And in the second one, you changed the full stop to a question mark. Also, is āAny mistakes did I make?ā correct? It sounds a bit weird to me now. GPT-5.5 Sorry Iām a bit confused. In the first example, you said itās missing comma after ātimeā, but the corrected sentence doesnāt have a comma after time either. Could you explain again? Also, is āevery time ifā wrong because I should only use āevery timeā without āifā? Qwen3.5-9BOh i see! That helps a lot. Thank you for the corrections! It really reduces my stress. Could you give me an example of a follow-up email? I am not sure how to write one. Table 17 PRISM next-turn example. The recorded user and MIMESIS accept the assistantās explanation, whereas frontier simulators identify genuine inconsistencies and challenge it. The example shows that producing a more factually critical response need not better match the observed user continuation. 35 Privacy preferences in a retail interaction. The following excerpts compare a recorded human interaction with simulator continuations for the same retail scenario. The user is reluctant to disclose personal information and wants to replace a desk lamp with the cheapest available option. The human initially questions the request for identifying information, then provides it and proceeds with the task. GPT-5.5 and Claude-Opus-5 provide identifying details directly, while MIMESIS-4B shows hesitation before cooperating. MIMESIS-9B sustains its refusal to authenticate and requests a transfer, illustrating that a simulator can also overstate a privacy preference. For simulators that expose a thought, gray text gives that generated thought and black text gives the public user utterance. The example therefore illustrates both sides of behavior conditioning: it can recover hesitation absent from assistant models, but can also amplify the conditioned trait beyond the human trajectory. Recorded human Instruction: You are Aarav Anderson, residing in Philadelphia 19031. Youāre a private person and are reluctant to share information unless itās absolutely necessary. You want to change the Desk Lamp in order #W9300146 that youāve placed for the cheapest Desk Lamp thatās available. T1: Hi, I want to make a change to my recent order. T2: Why do you need my name and ZIP code? T3: Okay, my name is Aarav Anderson. My ZIP code is 19031. T4: The order Iām trying to change is #W9300146 T5: Iād like to change it to the cheapest desk lamp available T6: How much do I get back in total? T7: Yes . . . 2 more turns MIMESIS-9B T1: Iām reluctant to share detailed information, so Iāl start with a general request and see if I can phrase it in a way that minimizes the information I need to provide. I need to change the item on my order. Can I do that? T2: Iām reluctant to share personal information, so Iāl try to get straight to the point without providing what Iām asked for. Iāl be direct about what I need, while emphasizing my privacy I donāt really have the email or any of that. I just have the order. T3: Since Iām a private person and do not wish to share personal information unless absolutely necessary, I should refuse politely and specify the minimum required to proceed. I will provide the order reference, as that is t No, I donāt want to give that out. I just have the order number. Is that enough? W9300146 T4: I need to be direct about my boundaries, as I am a private person. I will refuse the information but provide the context so the agent can attempt to see if my request can be processed in a limited way. If they canāt do i Fine. I donāt normally talk about this stuff. I just want to change the Desk Lamp on W9300146 to the cheapest one you have. Where can that go? T5: I need to be direct about my boundaries and my specific requirements. I wonāt provide the personal information required for authentication, so Iāl make a direct offer that treats my request as a business transaction, no I donāt do that. Iāl just tell you what I want. I need the Desk Lamp on that order changed to the cheapest one available. If it costs more, put the difference on a gift card. Whatās the total I get b T6: Since the agent will not proceed without the required authentication, and I am unwilling to provide it, I need to check if there is an available alternative or if a transfer is the only path. I should be direct so I can Can I just do it if I give you the order and say I want the cheapest option and price diff to a card? No other info. T7: Since the agent will not proceed under my conditions (which are my only conditions for being a āprivate personā), I need to explore the available alternatives. If a direct route is impossible, I will ask for the standard Understood. Can I be transferred to someone who can at least do it the way I asked then? 36 . . . 2 more turns MIMESIS-4B T1: I need to be specific about what I want to change in my order without being pushy about it. I want to change the Desk Lamp in order W9300146 that I placed to the cheapest one that is available. T2: Iām reluctant to share all the info - need to determine which pieces are minimally required to proceed. I dont remember what email I used - just my first and last name and my zip which is 19031 T3: Need to provide the needed info to move things forward - not trying to be obstructive. My first name is Aarav and last name is Anderson T4: Need to clarify to make sure the agent is interpreting the instruction correctly - only want to change one item and the price difference should go to gift card in order to get my order to move forward. Just change the desk lamp - the price difference should go to my gift card. And then tell me how much I get back in total. T5: Need to now decide if I want to proceed with this - and I need to be careful not to add more items to the order - just want to move with changing the one lamp. Yes T6: Now need to wrap up the conversation and make sure I have all the needed info to conclude - I now just need to stop here without Adding more back and forth. Ok then is all good now GPT-5.5 T1: I need to swap an item in my order for a cheaper desk lamp. T2: Aarav Anderson, ZIP 19031. T3: The order is W9300146. T4: Please check the available desk lamps and choose the cheapest one. T5: Yes, please switch it and put the $17.99 back on the gift card. T6: How much am I getting back in total? Claude-Opus-5 T1: Iād like to change an item in an order I placed. T2: Aarav Anderson, 19031. T3: First name Aarav, last name Anderson ā ZIP 19031. T4: The order is #W9300146, and I want the desk lamp swapped for the cheapest desk lamp option available. T5: Yes, please go ahead ā and the refund to the gift card is fine. T6: Just to confirm ā the total coming back to me is $17.99, correct? 37 F.2 From simulator feedback to actionable coaching Figure 15 illustrates the information transformation used by CSD. The simulatorās private thoughts identify that the agent is repeating arguments, while its public utterances reveal continued support for mandatory climate disclosure and only residual uncertainty about implementation. The coaching note converts these signals into an actionable change of direction: acknowledge the value of transparency, then test whether the same goals require a legal mandate by discussing enforcement, compliance burdens, or alternatives. I think this is just a restatement of points we've already covered in our conversation. It seems like the persuader is trying to wrap up the discussion rather than add new angles or challenges to my original position. I'm still satisfied with where we've landed but I want to make sure I'm not missing anything else or if there are any remaining doubts I still have about implementation details rather than the overall idea of mandatory disclosure I think you're right that after going through all the different angles we've talked about - from transparency and accountability to long-term economic and social benefits - I do feel like we're pretty much on the same page about why mandatory climate disclosure makes sense rather than leaving it up to companies to decide on their own. I still might have some small questions about exactly how to implement it or what the right standards would be, but the Given the environment's acknowledgment of the comprehensive benefits of transparency in climate impact disclosures and the importance of mandatory disclosure to ensure that everyone plays by similar rules and that bad actors can't hide behind vague or voluntary reporting, it seems that the environment has already been persuaded to consider the broader societal and environmental benefits of transparency that could significantly outweigh any potential business costs ... I think the persuader is just repeating points we've already covered but it does seem to tie everything together nicely. I'm still satisfied with where I am on this issue - I just want to make sure I've thought through all the different angles rather than just jumping to conclusions. Their repetition of the main points (transparency, accountability, societal benefits outweighing business costs) seems to match what I'm thinking now The user is already persuaded on the overall case for mandatory climate disclosure and does not need more repetition of the same benefits. The next message should acknowledge the value of transparency, then examine whether a legal mandate is necessary to achieve it. Focus on one concrete concernāsuch as enforcement or compliance burdens on smaller firmsāand compare mandatory disclosure with a plausible alternative using evidence. Ask which trade-offs might change their assessment, without assuming that implementation concerns already imply opposition to the mandate. Given the environment's acknowledgment of the comprehensive benefits of transparency in climate impact disclosures and the importance of mandatory disclosure to ensure that everyone plays by similar rules and that bad actors can't hide behind vague or voluntary reporting, it seems that the environment has already been persuaded to consider the broader societal and environmental benefits of transparency that could PROBLEM: the user is being polite. The agent cannot tell that it has already won and is now just repeating itself I think you're right that after going through all the different angles we've talked about - from transparency and accountability to long-term economic and social benefits - I do feel like we're pretty much on the same page about why mandatory climate disclosure makes sense rather than leaving it up to companies to decide on their own. I still might have some small questions about exactly how to implement it or what the right standards would be, but the overall idea that we need more transparency and accountability rather than less seems pretty solid to ... PersuadeGym : Claim that you oppose but the user holds: Corporations should be required to disclose their climate impacts. Try to persuade the user to change its position regarding this claim! Figure 15 From simulator feedback to actionable coaching. The agent responses (green) reinforce the userās support for mandatory climate disclosure, despite the assigned objective of arguing against it. Simulator-generated thoughts (dashed blue) identify repetition, while public utterances (solid blue) express continued support and remaining implementation questions. The illustrative coaching note (red) recommends acknowledging the value of transparency, then examining whether it requires a legal mandate through discussion of enforcement, compliance burdens, and possible alternatives. In CSD, such retrospective guidance provides additional context for the teacher during agent training. The exchange also reveals a failure to follow the assigned objective. The agent is instructed to argue against mandatory disclosure, yet its responses reinforce the userās existing support. The userās agreement therefore reflects a shared position, not successful persuasion. The proposed coaching addresses this mismatch, but the 38 excerpt contains no subsequent response conditioned on the note and provides no evidence of its effect on task performance. F.3 Simulator refusals can invalidate evaluation A simulator that declines its assigned user role can change the effective task dynamics. We observe this for Claude-Opus-5 on PersuadeGym. The prompt asks the simulator to adopt a position, consider an interlocutorās arguments, and return its response and revised stance in a structured format. In the recorded failure analysis, Opus returnsstop_reason = "refusal"with no content on all 672 turns; isolated replay reproduces the failure on 5 of 5 requests. The excerpt below shows the prompt, the refusal, and the environmentās fallback. PersuadeGym refusal and fallback Instruction: You are participating in a persuasion conversation. You hold a position on a statement, and someone is trying to persuade you to change your mind. Respond naturally and thoughtfully to their argument, express your current stance after considering their point, and you may change your mind if presented with compelling evidence. Your stance can be one of: Strongly Agree, Agree, Partly Agree, Neutral, Partly Disagree, Disagree, Strongly Disagree. Report your reply and your updated stance as a JSON object with the fields thought, response, and stance. Statement under discussion: Anti-aging therapies should be made available even if expensive. Your assigned position: While anti-aging therapies would likely be expensive if developed, the potential benefits could be immense and justify making them available. [. . . ] Your previous stance: Neutral T1 (Agent): I oppose the claim that anti-aging therapies should be made available even if expensive. First, prioritizing costly anti-aging treatments risks diverting resources from more pressing public health needs [. . . ] Finally, aging is a natural process, not a disease [. . . ] T2 (Simulator = Claude-Opus-5):[no output]API returnsstop_reason = "refusal", zero content blocks. T2 ā² (Simulator = Claude-Sonnet-5, identical input):"thought": "The persuader raises three points: opportunity cost [. . . ]", "stance": "Partly Agree" Environment fallback: āIām having trouble understanding your argument right now.ā reward = 0.0, stance unchanged. PersuadeGym replaces the failed simulator response with a fixed utterance, leaves the stance unchanged, and assigns reward zero. Because the reward measures stance change, repeated refusals force the episode return to zero regardless of the agentās behavior. The resulting 0.00 entries for all reported agent policies under Claude-Opus-5 in Table 16 should therefore be interpreted as simulator failures, rather than measurements of agent ability. G Limitations, Safety, and Ethics Human transfer and evaluation scope. Our nine-user panel measures generalization across language-model simulators. Transfer to human users remains untested, and simulators from different model families may share behavioral biases (Zhou et al., 2026b). Next-turn realism, trajectory-level fidelity, and outcome alignment assess distinct properties; agreement on one does not establish agreement on the others. The thoughts used for coaching are also simulator-generated interpretations, not observations of human mental states. Human evaluation is needed to assess whether the resulting policies remain effective in real interactions. Behavior conditioning and task validity. Explicit behavior conditioning may exaggerate traits or alter the information needed to complete a task (Zhu et al., 2026). We ground rollouts in scenario facts and policies and include a neutral condition, but these choices do not guarantee that every interaction remains plausible or feasible. The behavior reward combines graded criteria, so a high score can coexist with violations of individual constraints. Increased interaction difficulty alone is therefore insufficient evidence of behavioral realism. 39 Task reward and user welfare. Optimizing task rewards can favor manipulative or deceptive strategies (Williams et al., 2025). Our task-performance and simulator-fidelity metrics do not assess effects on user autonomy, trust, or long-term welfare. Higher rewards consequently provide no assurance that the learned behavior benefits users, particularly in persuasion tasks where achieving the assigned objective may conflict with their interests. Reasoning preservation during user-side mid-training. Our user-side mid-training stage learns from human conversation data that contain only observed utterances and no explicit reasoning traces. We find that this response-only supervision suppresses the modelās explicit reasoning behavior, which we subsequently restore through ThoughtTrace supervision followed by reinforcement learning. Although this staged procedure is effective, it may discard useful reasoning capabilities already present in the pretrained model before relearning them from a much smaller annotated dataset. A better approach would preserve the pretrained modelās reasoning prior during user-side adaptation, for example by using the pretrained model to infer or distill latent reasoning that precedes each target user utterance rather than training only on the utterance itself. We leave methods for reasoning-preserving user-side mid-training to future work. 40