Paper deep dive
From Passive Delegates to Strategic Negotiators: Reinforcing Social Reasoning in Small Language Models with SocialRL
Wenyue Hua, Zachary Huang, Tyler Payne, Safoora Yousefi, Saleema Amershi, Asli Celikyilmaz
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/17/2026, 4:22:29 AM
Summary
This paper introduces SocialRL, a training framework and infrastructure for enhancing social reasoning in small language models (4B parameters) to act as strategic delegates. The authors train models across six negotiation domains (Deal-or-No-Deal, CaSiNo, Craigslist, Job Interview, Calendar, Marketplace) using a decoupled, event-based environment interface. Results show that domain-trained 4B models match or exceed GPT-5 family performance in specific domains. Furthermore, by consolidating specialists via cascade RL and multi-teacher on-policy distillation (OPD), a unified 4B model achieves an average utility of 0.627, surpassing GPT-4.1, GPT-5.1, and GPT-5.2. The study also highlights the importance of Theory-of-Mind (ToM) scaffolding, specifically next-action prediction, for improving generalization.
Entities (16)
Relation Signals (16)
Unified 4B Model ā achievesutility ā 0.627
confidence 95% Ā· unified 4B that reaches 0.627 average utility across all six environments
SocialRL ā appliedto ā Deal-or-No-Deal
confidence 95% Ā· apply it to a 4B model across six domains: Deal-or-No-Deal
SocialRL ā appliedto ā CaSiNo
confidence 95% Ā· apply it to a 4B model across six domains: ... CaSiNo
SocialRL ā appliedto ā Craigslist
confidence 95% Ā· apply it to a 4B model across six domains: ... Craigslist
SocialRL ā appliedto ā Job Interview
confidence 95% Ā· apply it to a 4B model across six domains: ... Job Interview
SocialRL ā appliedto ā Calendar
confidence 95% Ā· apply it to a 4B model across six domains: ... Calendar
SocialRL ā appliedto ā Marketplace
confidence 95% Ā· apply it to a 4B model across six domains: ... Marketplace
SocialRL ā trains ā 4B Model
confidence 95% Ā· We present SocialRL, a general recipe that trains social reasoning directly, and apply it to a 4B model
On-Policy Distillation ā ā
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:AI agents increasingly act on their users' behalf, handling tasks such as scheduling meetings, comparing offers, and haggling over prices. These principal-driven tasks routinely place the agent across from a counterpart (another user's agent, a seller, a recruiter) whose goals may conflict with its principal's. Yet the dispositions that make an assistant pleasant can make it a poor delegate: a friendly, helpful frontier model may disclose its principal's private information unprompted and concede at the first sign of resistance. We present SocialRL, a general recipe that trains social reasoning directly, and apply it to a 4B model across six domains: Deal-or-No-Deal, CaSiNo, Craigslist, Job Interview, Calendar, and Marketplace. Every domain is trained in-domain under the same recipe, and every policy is evaluated on all six. We find that (1) in-domain training reaches the frontier: on held-out scenarios the 4B matches or exceeds the GPT-5 family per domain, closing 73-122% of the baseline-to-frontier gap on the negotiation games, with 78% of buyer openings anchoring below target versus 3% untrained; (2) cross-domain transfer follows game structure: structurally paired games lift each other, a broad multi-issue donor lifts nearly all domains, and structurally isolated games transfer nothing; (3) guided by this transfer structure, two strategies, cascade RL and multi-teacher on-policy distillation (OPD), consolidate the per-domain specialists into a single unified 4B that reaches 0.627 average utility across all six environments, matching or exceeding GPT-4.1 (0.625), GPT-5.1 (0.619), and GPT-5.2 (0.613); (4) an explicit theory-of-mind scaffold helps only through training: distilling the ToM trace, rather than actions alone, lifts utility on every environment and generalizes better across them, and of the two ToM skills, only next-action prediction predicts negotiation outcomes.
Tags
Links
- Source: https://arxiv.org/abs/2608.13787v1
- Canonical: https://arxiv.org/abs/2608.13787v1
Trouble viewing inline? Open PDF directly ā
Full Text
109,968 characters extracted from source content.
Expand or collapse full text
From Passive Delegates to Strategic Negotiators: Reinforcing Social Reasoning in Small Language Models with SocialRL Wenyue Hua Thanks: Corresponding authors: wenyuehua@microsoft.com, aslic@microsoft.com, samershi@microsoft.com Affiliation: Microsoft Research, AI Frontiers Zachary Huang Affiliation: Microsoft Research, AI Frontiers Tyler Payne Affiliation: Microsoft Research, AI Frontiers Safoora Yousefi Affiliation: Microsoft Research, AI Frontiers Saleema Amershi Affiliation: Microsoft Research, AI Frontiers Asli Celikyilmaz Affiliation: Microsoft Research, AI Frontiers Abstract AI agents increasingly act on their usersā behalf as representatives of their interests, handling tasks such as scheduling meetings, comparing offers, and haggling over prices. These principal-driven tasks routinely place the agent across from a counterpart, such as another userās agent, a seller, or a recruiter, whose goals may conflict with those of its principal. Yet the dispositions that make an assistant pleasant can make it a poor delegate: a friendly and helpful frontier model may disclose its principalās private information unprompted, and concede the principalās position at the first sign of resistance. We present SocialRL, a general recipe that trains social reasoning directly, and apply it to a 4B model across six interaction domains: Deal-or-No-Deal, CaSiNo, Craigslist, Job Interview, Calendar, and Marketplace. Every domain is trained in-domain under the same recipe, and every trained policy is evaluated on all six domains. We find that (1) in-domain training reaches the frontier: on held-out scenarios the 4B matches or exceeds the GPT-5 family per domain, closing 73ā122% of the baseline-to-frontier gap on the negotiation games, with the change visible at the trace level where 78% of buyer openings anchor below target versus 3% untrained; (2) cross-domain transfer follows game structure: structurally paired games lift each other, a broad multi-issue donor lifts nearly all domains, and structurally isolated games transfer nothing; (3) guided by this transfer structure, we propose two strategies to consolidate the per-domain specialists into a single unified 4B that achieves 0.627 average utility across all six environments, matching or exceeding GPT-4.1 (0.625), GPT-5.1 (0.619), GPT-5.2 (0.613): cascade RL and multi-teacher on-policy distillation (OPD); (4) an explicit theory-of-mind (ToM) scaffold helps only through training: distilling the ToM trace, rather than actions alone, lifts utility on every environment and generalizes better across them, and of the two ToM skills, only next-action prediction predicts negotiation outcomes. 1 Introduction AI agents are no longer only tools for completing tasks. A growing share of what they are asked to do carries the userās stake: the agent acts on the userās behalf, and the outcome bears directly on the userās interests (37; 35). Deployed systems can manage customer-service requests, coordinate meetings, purchase goods and book reservations, and help users compare homes or schedule property tours (23; 14; 33; 58). These applications position an AI agent as a delegate: a system entrusted with a principalās preferences and authorized to take consequential actions in open-ended environments. Delegation becomes especially challenging when an agent interacts with another party whose objectives differ from those of its principal. A customer-service agent (6) may face a user seeking compensation, a purchasing agent may negotiate with a seller seeking a higher price (15), and a scheduling agent (59) may need to reconcile participants with competing constraints. In such settings, merely completing the interaction is insufficient. A capable delegate must protect private information (20), infer the counterpartās incentives (30), determine when to concede or push back, and pursue an agreement that advances its principalās interests (36). We refer to this collection of capabilities as social reasoning: reasoning about another actorās latent preferences, incentives, and likely behavior in order to choose strategically effective actions. Figure 1: SocialRL overview: interaction environments, event-based agent interface, and decoupled training infrastructure. (a) Example social-reasoning environments for delegated coordination and negotiation. (b) An event-based, agent-independent interface in which a stateful environment communicates with agents through private channels of observations, notifications, and actions. (c) A decoupled training stack where the rollout proxy connects the environment and agent harness to inference and training, enabling interchangeable components and supporting both reinforcement learning and distillation. Negotiation has long served as a testbed for language-based strategic interaction, including multi-item allocation, resource division, and price bargaining (24; 16; 7). Recent work has extended these settings to evaluate the agency and strategic behavior of general-purpose language models (5; 53; 17). Although frontier models can conduct coherent negotiations and frequently reach agreements, strong aggregate outcomes do not guarantee faithful representation of the principal. We find that frontier models routinely abandon contested positions after limited resistance. These behaviors reflect dispositions that are useful for general assistance, such as transparency, agreeableness, and an eagerness to reach consensus, but they leave an agent predictable and exploitable when acting as a delegate. These failures reflect a mismatch between the objectives used to train general-purpose assistants and those required for strategic delegation. Instruction tuning and reinforcement learning from human feedback are typically designed to produce broadly helpful (8), honest, harmless, and instruction-following behavior in cooperative dialogue (21; 54; 9). In interactions with conflicting objectives and private information, the same behavioral priors can manifest as premature disclosure, excessive accommodation, and a preference for agreement even when the resulting outcome is unfavorable to the principal. Delegated agency therefore requires principal-conditioned strategic behavior: selective information disclosure, calibrated reservation boundaries, and the willingness to reject or prolong an interaction when agreement would sacrifice the principalās utility. This objective mismatch motivates post-training that targets social reasoning directly. We present SocialRL, a complete infrastructure + recipe for training and studying social reasoning in language-model agents, as shown in Figure1. We focus on a 4B model to test whether strategic delegation can be induced through targeted post-training rather than depending on frontier-scale capacity. Using SocialRL, the resulting 4B policies approach and, in some settings, exceed the performance of much larger GPT models while learning strategies that generalize across interaction structures. We train and evaluate across six interaction domains: Deal-or-No-Deal (24), CaSiNo (7), Job Interview (44), Craigslist (16), and Calendar and Marketplace from SocialReasoning-Bench (31). Together, these domains span single- and multi-issue bargaining, price negotiation (57), and slot coordination. To support training in these multi-turn environments, we build a decoupled system (52; 40; 27) that separates the environment, agent harness, inference engine, and trainer, allowing a common interaction infrastructure to support reinforcement learning, distillation, and heterogeneous or black-box counterparts. We first train a specialist within each domain and evaluate the full cross-environment transfer matrix, then consolidate the specialists into a single policy across all six environments using cascade RL (10) as well as multi-teacher mode-seeking OPD. On held-out scenarios, our domain-trained 4B policies achieve performance comparable to much larger GPT models across the six environments. Cross-environment transfer is substantial and follows interaction structure, with the strongest transfer occurring between structurally similar domains. We consolidate these specialized policies through cascade reinforcement learning and multi-teacher OPD (MOPD), producing a unified 4B model that achieves 0.627 average utility across all six environments, comparable to GPT-4.1 at 0.625, GPT-5.1 at 0.619, GPT-5.2 at 0.613. Finally, we introduce explicit theory-of-mind supervision through Infer ā Act ā Anticipate. Distilling these reasoning traces improves performance and cross-environment generalization, with next-action prediction emerging as the theory-of-mind skill most predictive of negotiation outcomes. Contributions. ā¢ Strategic Delegation Formulation & Six-Environment Suite. We formulate strategic delegation as a social-reasoning post-training problem and introduce a heterogeneous suite spanning allocation, multi-issue bargaining, price negotiation, and preference-based coordination. We train domain specialists in each environment and evaluate every policy across the full cross-environment transfer matrix. ā¢ Agent-Agnostic Environments & Decoupled Training Infrastructure. We develop SocialRL, combining an event-based, agent-independent environment interface with an OpenAI-compatible rollout proxy that disentangles environments, agent harnesses, inference engines, and trainers. The same stack supports reinforcement learning and distillation with local, remote, heterogeneous, or black-box counterparts. ā¢ Frontier-Range 4B Policies & Transfer-Aware Unification. We show that domain-trained 4B policies achieve aggregate performance in the range of GPT-4.1, GPT-5.1, and GPT-5.2, and reveal that cross-environment transfer is strongly directional and structure-dependent. We exploit this transfer structure to consolidate the specialists: transfer-aware cascade RL reaches 0.6270.627 Avg-6, while multi-teacher on-policy distillation recovers 92.6%92.6\% of the specialistsā average advantage in only 6060 additional optimization steps. ā¢ Explicit Theory-of-Mind Supervision. We introduce Infer ā Act ā Anticipate supervision and show that distilling the complete reasoning trace outperforms action-only supervision on every evaluated negotiation environment and improves cross-environment generalization. We further identify next-action prediction, rather than preference inference alone, as the theory-of-mind component most predictive of negotiation outcomes. 2 Related Work Language-model agents. A large body of work on language-model agents has focused on enabling models to reason, plan, and act through external tools and digital interfaces. ReAct (49) introduced the interleaving of language-based reasoning with environment actions, while Toolformer (34) demonstrated that language models can learn when and how to invoke external APIs. Subsequent benchmarks have evaluated increasingly realistic forms of web navigation, desktop control, and tool-mediated interaction, including WebArena, OSWorld, and Ļ-bench (55; 43; 48). This literature has substantially advanced planning (41), tool selection (46), interface grounding (3), and policy compliance (11), with performance generally measured by whether the agent reaches a target environment state. Strategic interaction introduces an additional dimension: the environment contains another adaptive decision-maker (2), so the value of an action depends on its immediate effect, the information it reveals, and the future responses it induces. LLMs as delegated agents. LLMs are increasingly studied as delegates that act on behalf of users or organizations in economically and socially consequential settings. Ļ-bench(48) models agents handling retail and airline customer-service requests under domain-specific policies, while ScheduleMe (2) applies multi-agent coordination to personal calendar management. CalBench further studies calendar assistants that coordinate under private information, exposing trade-offs between scheduling efficiency, fairness, communication, and privacy (59). In commerce, ACES evaluates agents that inspect marketplaces and make product choices on behalf of consumers (1); in negotiation, recent work has compared advisory, coaching, and autonomous delegation interfaces in multi-party bargaining. These settings require the agent to preserve and act on a principalās preferences while interacting with users, platforms, or other agents whose objectives may differ. Negotiation is a particularly direct instance of delegated agency because success depends jointly on reaching an agreement, managing private information, and securing value for the represented principal. Training agents for delegated interaction. A growing line of work trains language agents through strategic interaction. Within negotiation, prior work has explored self-play and language feedback, iterative self-play with behavior cloning, reinforcement learning with verifiable economic rewards, and pipelines combining supervised training on synthetic negotiations with reinforcement learning (13; 25; 26; 4). These methods can induce substantially stronger bargaining strategies, but typically specialize to a particular interaction structure, such as bilateral price bargaining or resource division. Complementary work has broadened the scope of interactive post-training: Sotopia-Ļ and Sotopia-RL train general social behavior from open-ended interactions (39; 50), while recent self-play methods train transferable multi-agent reasoning across cooperative and competitive games (51; 19; 28). Evaluation frameworks have likewise highlighted the diversity of negotiation itself, spanning resource allocation, exchange, price bargaining, and realistic multi-issue scenarios (5; 56). SocialRL focuses on the intersection of these directions: post-training a single small model across heterogeneous delegated interactions that demand different strategic capabilities but share a common need to reason about counterpart incentives and act on behalf of a principal. We study transfer across these interaction structures and show that the resulting 4B policies reach the performance range of much larger GPT models. 3 Environment and Infrastructure for SocialRL Training agents through negotiation requires infrastructure that differs from conventional single-turn language-model post-training. An environment may be stateful, partially observed, and populated by multiple agents acting on different schedules. Each episode contains several model calls, while the reward is often available only after the complete interaction. In addition, experiments may combine local trainable policies, remote frontier-model opponents, scripted agents, and human participants. We therefore design the system around two forms of decoupling: an event-based interface separates environments from agents, and an OpenAI-compatible proxy separates agent execution from model training. Figure 2 shows the environment stack, while Figure 3 shows the training data flow. 3.1 Environment and Agent Interface We design a generic interface for multi-agent interaction that supports sequential and simultaneous decisions, asynchronous communication, and arbitrary numbers of participants. The same abstraction can represent two-party negotiation environments such as Deal-or-No-Deal and Craigslist, as well as more complex multi-party environments such as Werewolf and Avalon. The design follows four principles: environments are independent of agent implementation; interaction is represented as an event stream with explicit action semantics; agents consume this stream asynchronously; and failures are handled without terminating the episode. Agent-independent environments. The environment is a stateful process that owns the game rules, hidden state, legal actions, and reward computation, while making no assumptions about how participants are implemented. Each participant interacts with the environment through a private asynchronous channel and is treated as a black box implementing a single operation, decide(context, actions) ā action. An agent may therefore be backed by a language model, an external coding agent, a scripted policy, or a human interface. We further separate the model layer from the agent harness: model inference exposes only standard chat messages and tool definitions through an OpenAI-compatible API. This separation allows the environment, agent implementation, model backend, and execution transport to vary independently. Event-based interaction and action semantics. Communication between the environment and an agent is represented as an ordered stream of events rather than repeated snapshots of environment state. We distinguish two event types: an observation records something that has occurred, while a notification requests a decision from the receiving agent and specifies its currently available actions. The environment drives interaction through three primitives: ask() for sequential decisions, ask_all() for simultaneous decisions, and broadcast() for events requiring no response. The control flow of the environment coroutine therefore expresses the game logic directly, without requiring a separate state-machine or dispatcher abstraction. An event stream preserves information that is important in partially observable multi-agent interaction: what happened, when it happened, and which participants observed it. It also naturally separates an agentās action from its effect. Submitting an action records a decision and returns only an acknowledgment or validation error; its consequences are emitted subsequently as observations. This distinction is important because actions need not map one-to-one to outcomes. An outcome may depend on several participantsā actions, occur only after additional environment logic, or be visible to different participants in different ways. For example, a vote is an individual playerās action, whereas an elimination is a collective outcome produced after all votes have been received. Partial observability is enforced at this same boundary. Each emitted event is filtered for its intended audience, so an agentās channel contains exactly the information available to that participant. The resulting event history can be shared by agent execution, player-facing interfaces, replay tools, and post-episode analysis, reducing the need for separate state projections or logs that may diverge from what the agent actually observed. Figure 2: Environment and agent architecture. A stateful environment communicates with black-box agents through per-agent asynchronous channels carrying structured observations, notifications, and actions. The same interface supports LLM agents, external coding agents, humans, and scripted policies. Model inference is isolated behind an OpenAI-compatible interface, while the event stream supports agent execution, human play, observability, replay, and trajectory collection. Asynchronous event consumption. For local execution, agents consume their event streams directly through asynchronous iterators. This interface matches the execution pattern of multi-agent interaction: events arrive incrementally, agents may spend different amounts of time computing their decisions, and progress may depend on responses from other participants. Asynchronous consumption allows agents and the environment to execute concurrently while preserving the ordering of events, without requiring polling or coupling agent execution to the environmentās control flow. Remote agents access the same logical stream through a cursor-based HTTP interface. Events are stored in an append-only history and indexed by sequence number, allowing a disconnected client to resume from its last cursor without losing observations. Local and remote execution therefore share the same interaction semantics and differ only in transport. Failure and concurrency handling. The interface is designed so that transient agent and transport failures do not invalidate an episode. Every submitted action identifies the notification to which it responds. If the environment has advanced while an agent was generating its response, the action is rejected as stale and the agent consumes the newer events before deciding again. Together with the append-only event history, this provides resumable event delivery while ensuring that at most one action is accepted for each decision point. Invalid actions, timeouts, and unavailable agents are handled explicitly by the environment. Invalid actions generate informative events and may be retried, while a timeout applies an inert default action and records the failure in the event stream. A malformed response, failed model request, or dropped connection therefore becomes part of the trajectory rather than terminating the episode. This property is particularly important for large-scale rollout collection, where a single interaction may involve several independently executing models and individual requests may fail or exceed their latency budgets. Figure 3: Decoupled training architecture. The agent runtime executes independently of the trainer and sends standard model requests through a rollout proxy. The proxy forwards requests to the inference engine while recording the authentic model inputs and outputs associated with each episode. Completed trajectories are written to a shared Parquet buffer consumed by interchangeable training backends, and updated policy weights are synchronized back to the inference engine. 3.2 Decoupled Training Infrastructure Our second design goal is to disentangle rollout generation from training. Agentic rollouts may involve arbitrary environment logic, agent harnesses, memory systems, tool use, and remote counterparts, while the trainer should only require the model inputs, model outputs, and eventual rewards needed to optimize the policy. We therefore connect rollout and training through a single abstraction: an OpenAI-compatible rollout proxy placed between the agent runtime and the inference endpoint. Rollout-training separation. The environment and agent harness execute exactly as they would at inference time. The environment determines observations, actions, and rewards; the agent decides how to construct prompts, manage memory, invoke tools, and interpret model outputs. Neither contains training-specific logic. From the trainerās perspective, these implementation details are invisible: it consumes trajectories captured at the model API boundary and therefore need not know how the environment or agent harness is implemented. This separation is particularly useful for agentic interaction, where a single episode may contain many model calls, the terminal reward may become available only after the interaction ends, and counterparts may be remote or black-box systems outside the training process. It also allows the rollout stack and training stack to evolve independently. The rollout proxy as the training boundary. The proxy exposes the same OpenAI-compatible interface as the underlying inference endpoint, so an agent can use it by changing only the clientās base URL. For every model call, the proxy records the exact request received from the agent and the exact response returned by the model, while forwarding the response to the agent unchanged. It therefore captures the authentic model inputs and outputs produced during execution, without requiring the harness to construct a separate training representation. Each episode is associated with a unique session key carried with every model request. The session groups otherwise stateless HTTP calls into a trajectory. When an episode begins, the runtime opens a session; once the interaction terminates, it attaches the final reward and closes the session. Many rollout workers can consequently execute concurrently against a shared continuously batched inference server without exposing their internal control flow to the trainer. Trajectory reconstruction and memory compaction. Importantly, the proxy does not assume that the agent maintains its context in any particular way. Agent harnesses commonly truncate histories, summarize earlier interactions, or otherwise compact memory during long rollouts. In such cases, a later model request may no longer extend the prefix of an earlier request. We handle this directly at the model-call boundary. Consecutive calls whose contexts share the expected prefix are treated as part of the same training segment. When this prefix relation breaks, the proxy starts a new segment. Thus, context truncation, summarization, and memory compaction require no special integration with the trainer: they simply induce multiple training segments within the same episode. Each segment retains the exact context under which its response was generated, avoiding any need to reconstruct an artificial conversation history that the policy never actually observed. A common contract for RL and distillation. The proxy supports both reinforcement learning and distillation while preserving the same rollout interface. For reinforcement learning, it records tokenized prompts and responses together with token-level log probabilities from the generating policy. Capturing these probabilities at generation time preserves the behavior policy required by PPO even if the policy is updated before optimization. For distillation, the proxy stores the raw messages, tool definitions, responses, and tool calls, allowing trajectories from closed-source or heterogeneous teachers to be tokenized later using the studentās tokenizer. Completed sessions are exported to a shared Parquet buffer that forms the data contract between rollout and training. RL records include response tokens, rollout log probabilities, rewards, task identifiers, and policy versions; distillation records preserve the raw interaction trace. Training losses are applied only to model response tokens. Because trainers consume this common representation, the rollout system can be paired with a lightweight local trainer, distributed PPO through veRL, alternative algorithms through phitrain, or external training backends through a small adapter. Asynchronous rollout and optimization. Rollout collection, optimization, and inference proceed asynchronously. Episode workers continuously generate trajectories and append completed sessions to the buffer; the trainer consumes them in batches, updates the policy, and synchronizes new weights to the inference server. The rollout and training systems can therefore scale independently. Asynchrony can introduce policy staleness when rollout generation runs ahead of optimization. We bound this effect with a capacity-based gate that limits the number of trajectories dispatched ahead of the trainer. Each trajectory is tagged with the policy version that generated it, while remaining off-policy differences are handled by the importance-sampling correction used during PPO. Together with the environment interface in Section 3.1, this design separates the major axes of an agent-training system: environments define interactions, harnesses define agent behavior, the proxy records authentic model calls, and trainers optimize over the resulting trajectories. Changing any one of these components does not require rewriting the others. 4 Social Reasoning Environments and Reward Design SocialRL trains agents across six heterogeneous negotiation and coordination environments: Deal-or-No-Deal, CaSiNo, Craigslist Bargains, Job Interview, Calendar, and Marketplace. Our training recipe proceeds in two stages. We first train a domain-specialized policy in each environment, then consolidate these specialists into a single policy that performs well across all six domains. We study two consolidation strategies in Section 5: cascade reinforcement learning, whose training order is chosen according to measured cross-domain transfer, and MOPD, which provides a substantially more efficient route to unification. The six environments are deliberately heterogeneous. They span multi-issue allocation, multi-issue contract negotiation, single-issue price bargaining, and preference-based coordination, with different action spaces, utility structures, information asymmetries, and interaction dynamics. Their common structure is social reasoning: in every domain, an agent must reason about a counterpart with private objectives, protect information about its own principal, and decide when to propose, concede, reject, or close. This diversity lets us study both specialization within individual interaction structures and transfer of social-reasoning strategies across them. Table 1 summarizes the six environments. Domain Structure Issues Private information Roles Deal-or-No-Deal multi-issue allocation 3 item types per-item values symmetric CaSiNo multi-issue allocation 3 resources priority order symmetric Craigslist single-issue price price target / listing price asymmetric Job Interview multi-issue contract 5 issues utility weights asymmetric Calendar slot coordination time slots slot preferences asymmetric Marketplace single-issue price price reservation price asymmetric Table 1: The six social-reasoning environments. The suite spans heterogeneous negotiation and coordination structures while sharing a common requirement to reason strategically about a counterpart with private objectives. 4.1 Interaction Environments Deal-or-No-Deal (DnD). Two agents divide a shared pool of items (books, hats, and balls), each holding private per-item values (24). A scenario specifies an item multiset and each playerās valuation, with every playerās total valuation normalized to 1010 points. Agents alternate free-form messages and structured proposals until a split is accepted, one side walks away, or the round limit is reached. Because valuations are private, effective play requires identifying which items the counterpart values and trading low-value items for high-value ones. CaSiNo. Two campers divide a fixed stock of food, water, and firewood packages, with three units of each resource (7). Each player has a private priority ordering over the resources, grounded in a persona and backstory. Priorities map to per-unit utilities (High =5=5, Medium =4=4, Low =3=3), giving each player a maximum score of 3636. CaSiNo shares DnDās multi-issue allocation structure while requiring agents to negotiate through naturalistic needs and justifications. Craigslist Bargains. A buyer and seller negotiate the price of a single listed item (16). Scenarios are derived from Craigslist postings across multiple product categories, with private price objectives defining the bargaining range. The environment is one-dimensional and role-asymmetric: the buyer seeks a lower price while the seller seeks a higher one. Effective negotiation therefore depends on anchoring, concession timing, and deciding when the available surplus justifies agreement. Job Interview. A worker and recruiter negotiate a five-issue employment package covering salary, weekly holiday, position, workplace, and company (44). Each side has private utilities and issue weights, and the parties generally value different dimensions of the contract. Efficient agreements therefore require identifying the counterpartās priorities and exchanging concessions on low-value issues for gains on high-value ones. With roughly 10410^4 possible agreements per scenario, Job Interview provides the largest structured outcome space in the suite. Calendar. Calendar is adapted from SocialReasoning-Bench (31). An agent acts on behalf of a principal to negotiate a meeting time with a requestor whose preferences differ from the principalās. Each side has private preferences over available time slots, and the initial request conflicts with the principalās interests. The agent must gather enough information about the requestorās flexibility while steering the interaction toward a slot favorable to its principal. Marketplace. Marketplace is also adapted from SocialReasoning-Bench (31). A buyer agent negotiates a purchase with a seller while holding a private reservation price. The seller begins from a price unfavorable to the buyer, while a zone of possible agreement is guaranteed to exist. As in Craigslist, success requires strategic price negotiation, but the environment differs in its interface, counterpart behavior, and utility construction, making the pair useful for studying transfer across structurally related domains. 4.2 Outcome Evaluation and Reward Design All six environments use terminal, outcome-only rewards: each episode receives a single scalar after agreement, walk-away, or timeout, with no intermediate reward shaping. Our reward design follows two principles. First, every environment maps outcomes to a common [0,1][0,1] range, providing consistent endpoints for cross-environment reporting, aggregation, and checkpoint selection during multi-domain training. Second, the reward should reflect the quality of an agreement relative to the opportunities available in the particular scenario and role. The appropriate normalization therefore depends on the interaction structure: DnD, CaSiNo, and Job Interview use scenario- and role-specific reference agreements; Craigslist and Marketplace normalize by the available price-negotiation corridor; and Calendar normalizes by the value range over mutually feasible time slots. For a fixed scenario and reference point m, the transformation is strictly increasing in z and therefore preserves the ranking of outcomes within that scenario. Its purpose is to calibrate how outcomes are valued across scenarios. For example, a normalized utility of 0.60.6 may exceed the fair-and-efficient reference in a highly conflicting scenario while falling well below it in a scenario with largely compatible preferences. Centering the reward curve at m assigns the greatest resolution to the transition between weak and strong agreements under the scenario-specific benchmark. Difficulty-aware rewards for multi-issue negotiation. DnD, CaSiNo, and Job Interview evaluate an outcome relative to a scenario-specific reference agreement. This adjustment accounts for variation in the utility that a player can reasonably attain under different combinations of private preferences. Allocation games. In DnD and CaSiNo, a scenario specifies item counts ck\c_k\ and private per-item values vav^a and vbv^b. For an allocation x, where player a receives xkx_k units of item k, the two players obtain saā(x)=ākxkāvka,sbā(x)=āk(ckāxk)āvkb.s_a(x)= _kx_kv^a_k, s_b(x)= _k(c_k-x_k)v^b_k. (1) The maximum raw score is S=10S=10 in DnD and S=36S=36 in CaSiNo. Because the allocation spaces are small, we enumerate all feasible outcomes and compute their Pareto frontiers exactly. We use allocations that are both Pareto-optimal and envy-free to define the reference point. An allocation is envy-free when each player weakly prefers its own bundle to the other playerās bundle under its private valuation. For player i, we define the envy-free Pareto maximum mi=1Sāmaxā”siā(x):xā is Pareto-optimal and envy-free.m_i= 1S \s_i(x):x is Pareto-optimal and envy-free \. (2) This reference is the greatest normalized value available to player i among outcomes that remain both efficient and fair. In scenarios for which no allocation satisfies both conditions, we use the Pareto-optimal allocation that minimizes the larger of the two playersā envy violations. Job Interview. Each Job Interview outcome is a deal d assigning one option to each issue. The worker and recruiter receive normalized utilities uwā(d),urā(d)ā[0,1]u_w(d),u_r(d)ā[0,1] derived from their private issue weights and option utilities. Because the complete outcome space is enumerable, we define the reference agreement through the egalitarian objective dā=argā”maxdā”minā”(uwā(d),urā(d)),d^*= _d (u_w(d),u_r(d) ), (3) breaking ties in favor of greater total utility. The role-specific reference points are mw=uwā(dā),mr=urā(dā).m_w=u_w(d^*), m_r=u_r(d^*). (4) Common difficulty-aware transformation. Let zā[0,1]zā[0,1] denote the agentās normalized outcome utility and mā[0,1]mā[0,1] its scenario- and role-specific reference point. DnD, CaSiNo, and Job Interview use r=Ļā”(zāmT)āĻā”(āmT)Ļā”(1āmT)āĻā”(āmT),Ļā”(x)=11+eāx,T=0.2.r= Ļ\! ( z-mT )-Ļ\! ( -mT )Ļ\! ( 1-mT )-Ļ\! ( -mT ), Ļ(x)= 11+e^-x, T=0.2. (5) The transformation preserves r=0r=0 at z=0z=0 and r=1r=1 at z=1z=1, while placing the steepest part of the curve at z=mz=m. Outcomes near the reference agreement therefore receive the greatest reward discrimination. The same raw utility can represent strong play in a highly conflicting scenario and a weak agreement in a scenario with largely compatible preferences. This normalization retains a common absolute reward range while accounting for what was reasonably attainable in each scenario and role. It therefore gives PPO a more comparable signal of agreement quality across scenarios with different utility frontiers. Price-corridor rewards for single-issue negotiation. Craigslist and Marketplace both measure how much of a one-dimensional bargaining surplus each party captures. Let pminp_ denote the buyer-favorable endpoint of the bargaining corridor, pmaxp_ the seller-favorable endpoint, and pdealp_deal the agreed price, with pmin<pmaxp_ <p_ . We define rbuyer=clipā”(pmaxāpdealpmaxāpmin,0,1),rseller=clipā”(pdealāpminpmaxāpmin,0,1).r_buyer=clip ( p_ -p_dealp_ -p_ ,0,1 ), r_seller=clip ( p_deal-p_ p_ -p_ ,0,1 ). (6) For any agreement within the corridor, the two rewards sum to 11. A deal at pminp_ assigns the full available surplus to the buyer, while a deal at pmaxp_ assigns it to the seller. In Craigslist, the corridor is defined by the buyerās target price ptp_t and the listing price pāp_ : pmin=pt,pmax=pā.p_ =p_t, p_ =p_ . (7) The resulting reward measures how much of the listing-to-target bargaining range each role captures. In Marketplace, the corridor is the zone of possible agreement defined by the sellerās and buyerās private reservation prices (31). Let psp_s be the sellerās reservation price and pbp_b the buyerās reservation price, with ps<pbp_s<p_b. We set pmin=ps,pmax=pb.p_ =p_s, p_ =p_b. (8) The trained Marketplace delegate represents the buyer, so its terminal reward is rbuyerr_buyer. A deal at the sellerās reservation price captures all available surplus for the principal and scores 11, while a deal at the buyerās reservation price captures none and scores 00. A deal above the buyerās reservation price also receives zero buyer value. Thus, Craigslist and Marketplace use the same normalized surplus-sharing reward and differ only in how the endpoints of the price corridor are specified: Craigslist uses the task-defined target and listing prices, while Marketplace uses the partiesā private reservation prices. Preference-based reward for slot coordination. In Calendar, the agent receives its principalās value function vā”(t)ā[0,1]v(t)ā[0,1] over candidate time slots (31). The requestor has a separate value function over the same slots, constructed in opposition to the principalās preferences. Let Z denote the zone of possible agreement, defined as the set of time slots that are mutually free on both calendars. We define the principal-best and counterparty-best feasible values as vmax=maxtāā”vā”(t),vmin=mintāā”vā”(t).v_ = _t v(t), v_ = _t v(t). (9) For a meeting scheduled at tdealāt_deal , the Calendar reward is rcalendar=vā”(tdeal)āvminvmaxāvmin.r_calendar= v(t_deal)-v_ v_ -v_ . (10) By construction, the feasible slots have different preference scores, so vmax>vminv_ >v_ . The principalās most preferred mutually feasible slot scores 11, while the slot most favorable to the requestor scores 00. This reward distinguishes agreement quality from task completion. Scheduling any mutually feasible meeting completes the coordination task, while the reward measures whether the selected slot actually advances the principalās preferences. Non-agreement outcomes and common scale. Timeouts and aborted interactions receive zero reward. Where an environment provides an explicit walk-away action, we retain its task-specific outside-option payoff. Calendar and Marketplace assign zero outcome value when no agreement is reached. Across all six environments, the resulting training signal lies in [0,1][0,1]. The environment-specific constructions preserve the relevant utility structure: fair and efficient allocation, division of price surplus, or preference-sensitive coordination, while the shared range makes reward statistics comparable and provides a consistent interface for the multi-domain consolidation methods in Section 5. 5 SocialRL Training and Unification Our study proceeds in two stages. In Stage 1, we train a domain-specialized policy for each of the six environments. PPO directly from the base model is sufficient for Deal-or-No-Deal, CaSiNo, Job Interview, and Calendar, while Craigslist and Marketplace use an SFT warm start before PPO. We then evaluate every specialist on all six environments, revealing substantial and highly asymmetric cross-environment transfer that depends on interaction structure (Ā§5.2). In Stage 2, we consolidate the specialists into a single multi-domain policy through two complementary approaches: transfer-aware cascade RL, which uses the observed transfer structure to prioritize final performance, and multi-teacher on-policy distillation (MOPD), which transfers most of the specialistsā advantage with substantially less additional training (Ā§5.3). As a complementary study, we introduce explicit theory-of-mind supervision through Infer ā Act ā Anticipate and test whether direct supervision of opponent modeling improves social reasoning and cross-environment generalization (Ā§5.4). All experiments use Qwen3-4B-Instruct-2507 as the base policy, providing a controlled test of whether targeted post-training can induce broad strategic capabilities in a compact model. Across our six-environment evaluation, the resulting specialists and unified policy achieve aggregate performance in the range of GPT-4.1, GPT-5.1, and GPT-5.2. 5.1 Stage 1: In-Domain Specialist Training We first train a specialized 4B policy for each of the six environments. For Deal-or-No-Deal, CaSiNo, Job Interview, and Calendar, we optimize the base model directly with PPO using the terminal outcome rewards defined in Ā§4.2. For the two price-negotiation environments, Craigslist and Marketplace, we initialize the policy with supervised fine-tuning (SFT) before applying PPO. This warm start places prerequisite bargaining behaviors within the policyās support, improving exploration; PPO then optimizes these behaviors for the environmentās terminal utility. All rollouts use the decoupled infrastructure described in Ā§3.2. Agents interact with their environments through the standard harness, while the rollout proxy records the model inputs, outputs, rollout log probabilities, and terminal rewards required for training. This separation allows the same PPO implementation to be used across environments with different interaction structures and agent implementations. PPO configuration. Unless otherwise stated, all PPO runs use a separate Qwen3-1.7B critic. We first warm up the critic for 3030 optimization steps while keeping the 4B actor frozen, and then jointly update the actor and critic. We use learning rate of 2Ć10ā62Ć 10^-6 for the actor with batch size 144, mini batch size 36, max gradient norm 5, and learning rate of 1Ć10ā51Ć 10^-5 for the critic. Max training steps is set to be 200. Under this configuration, the critic typically reaches an explained variance of approximately 0.80.8 before actor being trained, indicating that the value model captures most of the variation in episodic returns and provides a reliable baseline for PPO updates. DnD CaSiNo Craigslist Job Int. Calendar Mktplace Avg. Base 4B 0.583Ā±0.0010.583Ā± 0.001 0.476Ā±0.0140.476Ā± 0.014 0.318Ā±0.0120.318Ā± 0.012 0.479Ā±0.0320.479Ā± 0.032 0.301Ā±0.0170.301Ā± 0.017 0.174Ā±0.0260.174Ā± 0.026 0.389 Domain-trained 4B 0.656Ā±0.0080.656Ā± 0.008 0.503Ā±0.0070.503Ā± 0.007 0.583Ā±0.0080.583Ā± 0.008 0.594Ā±0.0070.594Ā± 0.007 0.540Ā±0.0170.540Ā± 0.017 0.838Ā±0.0130.838Ā± 0.013 0.619 GPT-4.1 0.653 0.491 0.540 0.588 0.673 0.804 0.625 GPT-5.1 0.671 0.499 0.607 0.596 0.573 0.767 0.619 GPT-5.2 0.663 0.488 0.577 0.579 0.643 0.727 0.613 GPT-5.5 0.665 0.559 0.746 0.590 0.702 0.985 0.708 Table 2: In-domain specialist training. Base and domain-trained 4B scores are reported as mean Ā± standard deviation across repeated evaluations; GPT baselines are reported as mean scores. Each domain-trained checkpoint is evaluated on its training environment. The six 4B specialists average 0.619 utility, placing their aggregate performance in the range of GPT-4.1, GPT-5.1, and GPT-5.2. As shown in Table 2, in-domain post-training produces strong specialists across all six interaction structures 11 1 We ran evaluation for 5 times to compute the mean and variance throughout the paper.. Averaged across domains, the specialized 4B policies achieve 0.615 utility, comparable to GPT-4.1 (0.625), GPT-5.1 (0.619), and GPT-5.2 (0.613). At the environment level, the 4B specialists fall within or above the GPT-4.1/5.1/5.2 range on DnD, CaSiNo, Craigslist, and Marketplace, while remaining close on Job Interview and Calendar. We next ask whether these domain-specific capabilities transfer across interaction structures. 5.2 Cross-Environment Transfer We next evaluate every domain-trained policy on all six environments, producing the transfer matrix in Table 3. The diagonal measures in-domain specialization, while each off-diagonal entry measures how training on one donor environment changes performance on another. We summarize each donor by its out-transfer, the mean change relative to the base model over the other five environments. Donor DnD CaSiNo Craigslist Job Int. Calendar Mktplace Out-tr. Baseline 0.583Ā±.0010.583Ā±.001 0.476Ā±.0140.476Ā±.014 0.318Ā±.0120.318Ā±.012 0.479Ā±.0320.479Ā±.032 0.301Ā±.0120.301Ā±.012 0.174Ā±.0030.174Ā±.003 ā Deal-or-No-Deal 0.656Ā±.008ā0.656Ā±.008 0.491Ā±.0120.491Ā±.012 0.282Ā±.0280.282Ā±.028 0.481Ā±.0160.481Ā±.016 0.370Ā±.0090.370Ā±.009 0.150Ā±.0030.150Ā±.003 +0.005+0.005 CaSiNo 0.597Ā±.0190.597Ā±.019 0.503Ā±.007ā0.503Ā±.007 0.309Ā±.0040.309Ā±.004 0.463Ā±.0160.463Ā±.016 0.331Ā±.0100.331Ā±.010 0.168Ā±.0220.168Ā±.022 +0.003+0.003 Craigslistā 0.564Ā±.0230.564Ā±.023 0.494Ā±.0070.494Ā±.007 0.583Ā±.008ā0.583Ā±.008 0.447Ā±.0110.447Ā±.011 0.266Ā±.0230.266Ā±.023 0.491Ā±.0070.491Ā±.007 +0.050+0.050 Job Interview 0.610Ā±.0170.610Ā±.017 0.511Ā±.0040.511Ā±.004 0.328Ā±.0100.328Ā±.010 0.594Ā±.007ā0.594Ā±.007 0.373Ā±.0220.373Ā±.022 0.161Ā±.0110.161Ā±.011 +0.026+0.026 Calendar 0.607Ā±.0100.607Ā±.010 0.474Ā±.0070.474Ā±.007 0.167Ā±.0130.167Ā±.013 0.311Ā±.0130.311Ā±.013 0.540Ā±.017ā0.540Ā±.017 0.172Ā±.0080.172Ā±.008 ā0.060-0.060 Marketplace 0.558Ā±.0100.558Ā±.010 0.464Ā±.0070.464Ā±.007 0.502Ā±.0150.502Ā±.015 0.371Ā±.0100.371Ā±.010 0.268Ā±.0150.268Ā±.015 0.838Ā±.013ā0.838Ā±.013 +0.001+0.001 GPT-4.1 0.653 0.491 0.540 0.588 0.673 0.804 ā GPT-5.1 0.671 0.499 0.607 0.596 0.573 0.767 ā GPT-5.2 0.663 0.488 0.577 0.579 0.643 0.727 ā GPT-5.5 0.665 0.559 0.746 0.590 0.702 0.985 ā Table 3: Cross-environment transfer. Each row evaluates a policy trained on one donor environment across all six domains (ā denotes the training domain). Out-transfer is the mean change from the base 4B policy over the five off-domain environments. ā Craigslist uses SFT+PPO. Reported uncertainties are standard deviations across three SGLang benchmark runs where available. Transfer depends strongly on the donor-recipient configuration. The clearest pattern is transfer between environments with similar interaction structure. Craigslist and Marketplace, both price-negotiation environments, exhibit the two largest cross-domain gains: training on Craigslist raises Marketplace from 0.1740.174 to 0.4910.491 (+0.317+0.317), while training on Marketplace raises Craigslist from 0.3180.318 to 0.5020.502 (+0.184+0.184). The allocation pair DnD and CaSiNo also transfers positively in both directions, although more modestly. Some environments transfer more broadly. Job Interview is the strongest general-purpose donor after Craigslist, with an average out-transfer of +0.026+0.026 and improvements on DnD, CaSiNo, Craigslist, and Calendar. In contrast, a policy can become very strong in-domain without becoming a useful donor: Marketplace reaches 0.8380.838 on its own task while its average effect on the other five environments is approximately neutral. Calendar is the clearest negative donor in some domains, with out-transfer of ā0.060-0.060, including large drops on Craigslist (ā0.1670.318\!ā\!0.167) and Job Interview (ā0.3110.479\!ā\!0.311). These results show that transfer is strongly asymmetric and structured: what matters is not simply how strong a specialist is, but which capability it learns and which environment receives it. This observation is central to unification. In the next stage, we exploit the measured transfer structure to choose the ordering of environments for cascade RL, and compare this transfer-aware ordering against random and anti-transfer orderings. 5.3 Stage 2: Consolidating Specialists into a Unified Model Stage 1 produces a strong specialist for each environment. Our next goal is to consolidate these capabilities into a single policy that performs well across all six interaction structures. This is non-trivial because training effects are coupled across domains: as shown by the transfer matrix in Table 3, optimizing one environment can improve, preserve, or interfere with performance on another. Sequential training can therefore both exploit positive transfer and induce catastrophic forgetting. We study two complementary approaches to unification, targeting different points in the performance-efficiency tradeoff. Two routes to unification. Our first approach, cascade RL (38; 10), continues reinforcement learning sequentially across environments. Because each stage directly optimizes environment reward, cascade RL can exploit positive transfer during training and may even improve previously learned capabilities beyond the corresponding single-domain specialists. Its cost is substantial: each environment requires another full RL stage, effectively relearning the capabilities rather than directly reusing the trained specialists. Moreover, the final policy depends strongly on training order, since beneficial and destructive transfer are highly asymmetric. We therefore explore the construction of a transfer-aware curriculum based on the transfer matrix from Section 5.2. Our second approach, MOPD (29; 45; 47; 12), reuses the Stage 1 specialists directly as teachers and distills their capabilities into a shared student. This is substantially more efficient: fewer than 100100 additional optimization steps recover most of the specialistsā advantage. The main challenge shifts from environment ordering to allocation of the distillation budget: different teachers provide very different amounts of additional capability over the student, and some have already been nearly matched through cross-domain transfer. We address this with a gap-closed curriculum that focuses sampling on domains whose teacher advantage remains unabsorbed. Shared initialization. Both consolidation methods start from the same Craigslist-SFT checkpoint. This initialization is necessary because the base 4B model rarely explores the anchoring behavior required for successful Craigslist negotiation: as observed in Stage 1, direct PPO fails to improve the base policy on Craigslist. We find the same limitation for OPD from the base model also fails to acquire the Craigslist capability when trained on Craigslist alone. Supervised distillation first places this prerequisite behavior within the studentās support, after which either RL or OPD can optimize and combine it with capabilities from the other environments. Using the same initialization for both methods also makes their comparison controlled. Cascade RL and MOPD thus provide complementary approaches to the same consolidation problem. Cascade RL prioritizes final performance by continuing reward optimization and exploiting cross-environment transfer, while MOPD prioritizes efficiency by directly transferring the capabilities already learned by the domain specialists. 5.3.1 Stage 2A: Transfer-Aware Unification with Cascade RL Training principle. Cascade RL consolidates capabilities by continuing reinforcement learning sequentially across environments. A single policy is carried through the cascade: at each stage, PPO optimizes the policy on one environment using the same outcome reward as in Stage 1, and the selected checkpoint initializes the next stage. We periodically evaluate checkpoints on all environments encountered so far and select the one with the highest average utility, balancing progress on the current domain against retention of previously acquired capabilities. The central design choice is therefore the order of environments. The transfer matrix in Table 3 shows that cross-environment effects are strongly directional: training on one domain may improve another, leave it largely unchanged, or substantially degrade it. Sequential RL is consequently path-dependent. We use the observed transfer structure to construct a curriculum that places beneficially interacting domains together and schedules destructive donors before the capabilities they would otherwise overwrite. Transfer-aware ordering. Three patterns in Table 3 guide the curriculum. First, Craigslist and Marketplace form the strongest transfer pair in the suite: Craigslist training improves Marketplace by +0.317+0.317, while Marketplace improves Craigslist by +0.184+0.184. We therefore place them consecutively, with Craigslist first to exploit the stronger transfer direction. Second, Calendar is a destructive donor for Craigslist and Job Interview: reducing Craigslist by 0.1510.151 and Job Interview by 0.1680.168, and is the only domain with negative average out-transfer. We therefore place Calendar at the beginning of the cascade, before these bargaining capabilities are acquired. Third, Job Interview exhibits strongly asymmetric transfer. It transfers positively to several other domains, yet its own performance is poorly preserved after training elsewhere. We therefore place it last, allowing the final stage to recover Job Interview capability without exposing it to subsequent interference. These considerations yield the transfer-aware curriculum CalendarāCaSiNoāDnDāCraigslistāMarketplaceāJob Interview.Calendar Interview. (11) CascadeRL configuration. All cascade experiments start from the same Craigslist-SFT initialization and use the same PPO configuration and per-stage training budget. Each stage is an independent PPO with max 200 steps, and we evaluate each checkpoint every 20 training steps on a small validation dataset. Ordering controls. To test whether the measured transfer structure provides a useful curriculum signal, we compare the transfer-aware ordering with alternative sequences while holding initialization and training budget fixed. Our random-order control uses CraigslistāJob InterviewāDnDāCaSiNoāCalendarāMarketplace.Craigslist Interview . (12) We additionally construct an anti-transfer curriculum that deliberately reverses the main ordering principles: Job InterviewāMarketplaceāCraigslistāDnDāCaSiNoāCalendar.Job Interview . (13) This ordering places the difficult-to-preserve Job Interview capability first and the destructive Calendar stage last. Unified-model performance. Table 5 reports the final six-environment performance of the transfer-aware, random-order, and anti-transfer cascades, together with the GPT baselines. The transfer-aware cascade reaches an Avg-6 of 0.627Ā±0.0040.627Ā± 0.004, placing the unified 4B policy in the same aggregate performance range as GPT-4.1 (0.6250.625), GPT-5.1 (0.6190.619), and GPT-5.2 (0.6130.613). In contrast, the random-order cascade reaches only 0.584Ā±0.0080.584Ā± 0.008 and the anti-transfer cascade reaches only 0.562Ā±0.0100.562Ā± 0.010 under the same overall training setup. The aggregate result masks substantial cross-domain interaction. Calendar is the clearest example of positive accumulation: although it is trained at the very beginning of the cascade, the final policy reaches 0.7420.742, well above its single-domain specialist score of 0.5400.540 and above every evaluated GPT baseline. Marketplace is also strongly retained at 0.8030.803, essentially matching GPT-4.1 at 0.8040.804, while Craigslist reaches 0.5800.580, above GPT-4.1 and GPT-5.2. Thus, sequential RL can do more than preserve earlier specialists: subsequent stages can reinforce capabilities acquired earlier through positive cross-environment transfer. Stage-by-stage transfer dynamics. The final benchmark does not reveal how capabilities evolve as the policy moves through the cascade. Table 4 traces the checkpoint selected at the end of each stage, evaluated on the environments included in checkpoint selection at that point on a small validation dataset. Because every cascade starts from the Craigslist-SFT initialization, Craigslist is tracked from the first stage even though its PPO stage occurs fourth. These values are the within-cascade evaluations used for checkpoint selection; the final repeated benchmark is reported separately in Table 5. Stage completed Step DnD CaSiNo Craigslist Job Int. Calendar Mktplace Mean seen Calendar 100 ā ā 0.614 ā 0.621 ā 0.617 CaSiNo 40 ā 0.504 0.517 ā 0.733 ā 0.585 Deal-or-No-Deal 100 0.618 0.498 0.489 ā 0.758 ā 0.591 Craigslist 40 0.628 0.477 0.544 ā 0.729 ā 0.595 Marketplace 140 0.628 0.483 0.564 ā 0.704 0.842 0.644 Job Interview 60 0.620 0.513 0.570 0.529 0.692 0.823 0.625 Table 4: Stage-by-stage evolution of the transfer-aware cascade. Each row reports the checkpoint selected after optimizing the environment in the first column; bold denotes the environment optimized at that stage. Craigslist is evaluated from the beginning because the cascade is initialized from its SFT checkpoint. Dashes denote environments not yet included in checkpoint selection. Mean seen averages the environments tracked at that stage and is therefore not directly comparable across rows as the set of environments expands. These are within-cascade checkpoint-selection evaluations; Table 5 reports the final repeated six-environment benchmark. The stagewise trajectory shows that cross-environment transfer occurs during the cascade itself. Calendar reaches 0.6210.621 after its own stage, then improves to 0.7330.733 after CaSiNo and 0.7580.758 after DnD despite receiving no additional Calendar-specific updates. Later bargaining stages introduce some forgetting, but Calendar remains strong at 0.6920.692 after the final Job Interview stage. Its high final performance therefore reflects capability accumulated across multiple stages, rather than retention of the original Calendar checkpoint alone. The Craigslist-Marketplace pair exhibits a second form of transfer. Craigslist performance initially falls during the allocation stages (0.614ā0.517ā0.4890.614ā 0.517ā 0.489), recovers to 0.5440.544 when Craigslist is optimized directly, and then rises further to 0.5640.564 after Marketplace training. At the same checkpoint, Marketplace reaches 0.8420.842 while DnD and Calendar remain at 0.6280.628 and 0.7040.704, respectively. The final Job Interview stage preserves most of these gains: Marketplace remains at 0.8230.823 and Craigslist rises to 0.5700.570, hile also raising CaSiNo from 0.4830.483 to 0.5130.513. These dynamics are neither monotonic nor uniformly positive. Earlier capabilities can temporarily degrade, as Craigslist does during the allocation stages, and Calendar declines from its intermediate peak as later domains are introduced. The role of the transfer-aware curriculum is therefore to order these interactions so that destructive stages occur early and later stages can repair or reinforce related capabilities. Checkpoint selection on the environments seen so far further limits forgetting at each transition. Environment Transfer-aware Random Anti-transfer Deal-or-No-Deal 0.617Ā±0.0080.617Ā± 0.008 0.604Ā±0.0120.604Ā± 0.012 0.622Ā±0.0180.622Ā± 0.018 CaSiNo 0.482Ā±0.0190.482Ā± 0.019 0.465Ā±0.0160.465Ā± 0.016 0.443Ā±0.0230.443Ā± 0.023 Craigslist 0.580Ā±0.0070.580Ā± 0.007 0.637Ā±0.0220.637Ā± 0.022 0.497Ā±0.0180.497Ā± 0.018 Job Interview 0.538Ā±0.0120.538Ā± 0.012 0.478Ā±0.0300.478Ā± 0.030 0.471Ā±0.0090.471Ā± 0.009 Calendar 0.742Ā±0.0370.742Ā± 0.037 0.541Ā±0.0210.541Ā± 0.021 0.641Ā±0.0110.641Ā± 0.011 Marketplace 0.803Ā±0.0080.803Ā± 0.008 0.783Ā±0.0240.783Ā± 0.024 0.698Ā±0.0220.698Ā± 0.022 Avg-6 0.627Ā±0.0040.627Ā± 0.004 0.578Ā±0.0080.578Ā± 0.008 0.562Ā±0.0100.562Ā± 0.010 Table 5: Cascade RL is sensitive to environment ordering. The transfer-aware curriculum follows Calendarā ā ā ā ā Interview; the random-order control follows Eq. 12; and the anti-transfer control follows Eq. 13. Each final evaluation uses 1010 scenarios Ć 33 opponents Ć 55 trials (150150 games per environment). Transfer-aware scores are mean Ā± standard deviation over two independent evaluations, and random-order scores are mean Ā± standard deviation over repeated evaluations. Anti-transfer performs way worse than both Transfer-aware as well as Random-order. The transfer-aware unified 4B policy achieves an Avg-6 of 0.6270.627, in the performance range of GPT-4.1, GPT-5.1, and GPT-5.2. Comparison with other curriculums. The comparison with random ordering and anti-optimal ordering shows that this accumulation depends strongly on curriculum. The transfer-aware cascade improves Avg-6 from 0.5840.584 to 0.6270.627, with particularly large gains on Calendar (0.541ā0.7420.541ā 0.742), Job Interview (0.478ā0.5380.478ā 0.538), and Marketplace (0.783ā0.8030.783ā 0.803). The random ordering performs better on Craigslist in isolation (0.6370.637 vs. 0.5800.580), but fails to preserve comparable performance across the full suite. This is precisely the objective of the transfer-aware curriculum: optimize the final multi-domain policy rather than any single stage. These results support the use of the transfer matrix as a reasonable curriculum-design signal. The same environments, trained with the same initialization and learning procedure, can lead to substantially different unified policies depending on their order. Cascade RL can therefore achieve strong multi-domain performance by exploiting the directionality of cross-environment transfer, at the cost of running a full sequence of reinforcement-learning stages. 5.3.2 Stage 2B: Efficient Unification with MOPD Training principle. Cascade RL continues optimizing environment rewards, but requires a full RL stage for each additional domain. MOPD instead treats the Stage-1 specialists as teachers and directly transfers their capabilities into a common student. At each rollout, we sample an environment e and pair it with its corresponding specialist. The student generates trajectories on-policy, and its responses are trained toward the teacher using a reverse-KL distillation objective. This construction preserves the original interaction stack: the environment and agent harness execute unchanged, while the rollout proxy records the student trajectory and queries the corresponding teacher for distillation. Changing from one teacher to six therefore requires no modification to the environments or agent implementations. MOPD configuration. The MOPD runs use a reverse-KL objective: the student generates every rollout on-policy, and the corresponding domain specialist supplies token-level teacher logits for the generated responses. We truncate the teacher distribution to its top 3232 logits, use a distillation temperature of 1.01.0, and apply the loss uniformly over all response tokens without position weighting. We optimize with a constant learning rate of 10ā510^-5 without warmup. Initialization and teacher selection. We initialize the unified student from the Craigslist-SFT checkpoint used throughout Stage 2. We also exclude teachers that offer essentially no additional headroom over this initialization. CaSiNo is the clearest case: the Craigslist-SFT student already scores 0.5020.502 on CaSiNo, nearly matching the domain specialist at 0.5030.503. Distilling from such a teacher provides little additional signal while potentially introducing optimization noise. We therefore omit CaSiNo from the MOPD training mixture and retain it only for evaluation. The transfer results in Table 3 further suggest that CaSiNo performance can be maintained through capabilities learned from other domains. Gap-closed curriculum. The remaining teachers still differ substantially in how much capability the student has left to absorb. Uniform sampling gives equal training budget to domains that are nearly solved and domains with large remaining gaps. We instead adapt the sampling distribution to the fraction of each teacherās advantage that remains unclosed: Let ā°E denote the candidate training environments. For each eāā°e , let BeB_e be the studentās pre-distillation score, TeT_e its specialist teacherās score, and sĀÆe(t) s_e^(t) the studentās rolling mean terminal reward over the most recent W=500W=500 training games. Before any observations are available, we initialize sĀÆe(t)=Be s_e^(t)=B_e. We define ge=TeāBe,Ļe(t)=sĀÆe(t)āBege,g_e=T_e-B_e, _e^(t)= s_e^(t)-B_eg_e, (14) where geg_e is the teacher-student headroom and Ļe(t) _e^(t) is the fraction already closed. The unnormalized sampling weight is we(t)=Ī»,ge<Ī“,maxā”(0,1āĻe(t))+Ī»,otherwise,w_e^(t)= casesĪ»,&g_e<Ī“,\\[4.0pt] \! (0,1- _e^(t) )+Ī»,&otherwise, cases (15) with floor Ī»=0.05Ī»=0.05 and minimum-gap threshold Ī“=0.03Ī“=0.03. We normalize pe(t)=we(t)āeā²weā²(t),p_e^(t)= w_e^(t) _e w_e ^(t), (16) and band-clamp the resulting probabilities to pe(t)ā[pmin,pmax],p_e^(t)ā[p_ ,p_ ], (17) iteratively renormalizing the remaining mass. pmin,pmaxp_ ,p_ are set to be [0.10,0.40][0.10,0.40] in our experiments. The normalization by each domainās own gap makes progress comparable across environments with different reward ranges. At initialization, Ļe=0 _e=0 for every learnable environment. As the student approaches its teacher, the corresponding sampling mass decreases; once it matches or surpasses the teacher, the environment receives only the floor weight. The minimum-gap guard prevents near-zero teacher-student gaps from turning reward noise into extreme sampling weights, while the probability band prevents any remaining environment from dominating or disappearing from training. Evaluation metrics. We evaluate every unified checkpoint on all six environments and report selected checkpoints as mean Ā± standard deviation over five independent evaluation runs. We use three complementary aggregate metrics. The first is the unweighted mean utility which directly measures overall model performance: Avg6ā”(s)=16āāese,Avg6(s)= 16 _es_e, (18) To measure the objective specific to specialist consolidation, we additionally compute the fraction of each teacherās advantage absorbed by the student: Ceā(s)=seāBeTeāBe.C_e(s)= s_e-B_eT_e-B_e. (19) A value of 11 means that the unified student matches the specialist teacher, and values above 11 indicate that it surpasses the specialist. CaSiNo is excluded because its teacher-initialization gap is only 0.0010.001. Finally, we report clipped gap closure where ā°ā²E excludes CaSiNo. CĀÆclip=1|ā°ā²|āāeāā°ā²clipā”(Ce,0,1), C_clip= 1|E | _e clip\! (C_e,0,1 ), (20) Clipping prevents over-performance on one environment from compensating for incomplete transfer on another. Avg-6 thus measures absolute capability, while the two gap-closure metrics measure how completely the specialist capabilities have been consolidated. Curriculum ablations. Table 6 compares adaptive and uniform environment sampling, with and without the saturated CaSiNo teacher. The gap-closed curriculum without CaSiNo performs best under all three metrics. After only 6060 optimization steps, it reaches an Avg-6 of 0.597Ā±0.0150.597Ā± 0.015, with 92.6%92.6\% mean gap closure and 84.5%84.5\% clipped gap closure. Removing CaSiNo helps under both uniform and adaptive sampling, confirming that allocating distillation budget to a teacher with essentially zero headroom is counterproductive. Configuration Avg-6 ā Gap closure ā Clipped gap closure ā Gap-closed, w/o CaSiNo 0.597Ā±0.0150.597Ā± 0.015 92.6Ā±11.392.6Ā± 11.3% 84.5Ā±6.384.5Ā± 6.3% Equal mix, w/o CaSiNo 0.594Ā±0.0200.594Ā± 0.020 84.4Ā±15.684.4Ā± 15.6% 79.7Ā±12.679.7Ā± 12.6% Equal mix, w/ CaSiNo 0.595Ā±0.0150.595Ā± 0.015 82.2Ā±9.282.2Ā± 9.2% 77.6Ā±5.877.6Ā± 5.8% Gap-closed, w/ CaSiNo 0.578Ā±0.0120.578Ā± 0.012 71.3Ā±8.771.3Ā± 8.7% 70.7Ā±7.970.7Ā± 7.9% Student initialization 0.460 ā ā Teacher average 0.619 ā ā Table 6: MOPD. All configurations use reverse-KL distillation without position weighting and a learning rate of 10ā510^-5. Scores are mean Ā± standard deviation over five independent evaluation runs. The gap-closed curriculum without the saturated CaSiNo teacher performs best under all three aggregate metrics. Per-environment consolidation. Table 7 shows how the best unified checkpoint compares with its initialization and specialist teachers. The unified model recovers 85%85\% of the Craigslist teacherās advantage, 69%69\% on Job Interview, and 83%83\% on Marketplace. It also surpasses the corresponding specialist on DnD (0.6640.664 vs. 0.6560.656) and Calendar (0.5640.564 vs. 0.5400.540), yielding gap closures above 100%100\%. Thus, MOPD can combine specialist capabilities without constraining the unified model to interpolate below each teacher. Environment Unified 4B Initialization Teacher Gap closed Deal-or-No-Deal 0.664Ā±0.0250.664Ā± 0.025 0.586 0.656Ā±0.0080.656Ā± 0.008 %111\% Craigslist 0.556Ā±0.0190.556Ā± 0.019 0.397 0.583Ā±0.0070.583Ā± 0.007 85%85\% CaSiNoā 0.478Ā±0.0190.478Ā± 0.019 0.502 0.503Ā±0.0080.503Ā± 0.008 ā Job Interview 0.551Ā±0.0230.551Ā± 0.023 0.456 0.594Ā±0.0070.594Ā± 0.007 69%69\% Calendar 0.564Ā±0.0500.564Ā± 0.050 0.375 0.540Ā±0.0170.540Ā± 0.017 %115\% Marketplace 0.771Ā±0.0280.771Ā± 0.028 0.444 0.838Ā±0.0130.838Ā± 0.013 83%83\% Avg-6 0.597Ā±0.0150.597Ā± 0.015 0.460 0.619 92.6%ā92.6\%^* Table 7: Per-environment performance of the best MOPD model. We report the gap-closed curriculum without CaSiNo at step 60. Unified-model scores are mean Ā± standard deviation over five independent evaluation runs. Gap closure measures the fraction of each specialistās advantage over the initialization recovered by the unified model. ā CaSiNo is excluded from MOPD training and gap-closure aggregation because its teacherāinitialization gap is only 0.0010.001. āMean gap closure over the remaining five environments. Overall, MOPD provides a substantially cheaper consolidation route than sequential RL: most of the specialist advantage is transferred in fewer than 100100 additional optimization steps. Together, the two Stage-2 methods expose a performance-efficiency tradeoff: cascade RL uses continued reward optimization and transfer-aware ordering to pursue a stronger final policy, while MOPD directly compresses the specialists into a unified model with a much smaller additional training budget. 5.4 Explicit Theory-of-Mind Supervision The training recipe so far improves negotiation through outcome-based reinforcement learning: the model is rewarded for reaching favorable agreements, while any reasoning about the counterpart is learned only indirectly. Yet effective negotiation depends naturally on understanding the other party (18; 32; 22; 42). An agent must infer what the counterpart values, identify where their preferences overlap or conflict, and anticipate how the counterpart is likely to respond to a proposed action. We therefore study a complementary approach that supervises these intermediate reasoning capabilities explicitly, asking whether stronger theory of mind (ToM) can translate into stronger negotiation. Theory-of-mind capabilities. We operationalize theory of mind through two capabilities that are directly relevant to strategic interaction. Preference inference (Infer) measures whether the agent can recover the counterpartās latent preferences or private state from the interaction history. Next-action prediction (Anticipate) measures whether the agent can predict how the counterpart will respond to the agentās planned move. Together with the agentās own action (Act), these form the reasoning scaffold InferāActāAnticipate. Infer\;ā\; Act\;ā\; Anticipate. (21) For each decision, the scaffold records the agentās current belief about the counterpart, its chosen action, and its prediction of the counterpartās subsequent reaction. We evaluate Infer by exact-match accuracy against the counterpartās private preferences and Anticipate by the accuracy of the predicted next action. Does prompting alone elicit theory of mind? We first test whether the base 4B model can benefit from the scaffold without additional training. The answer is negative: adding the explicit ToM prompt reduces average negotiation utility from 0.4540.454 to 0.3530.353 (Table 8), with the largest drop on Job Interview (0.476ā0.2210.476ā 0.221). Requiring a small model to produce an explicit reasoning structure therefore does not by itself improve strategic behavior. Direct evaluation of the two ToM capabilities helps explain this result. The base model already performs reasonably well at preference inference, reaching 0.6180.618 accuracy, but is substantially weaker at next-action prediction, at approximately 54%54\%. Thus, the base model can often recover what the counterpart wants while remaining much less capable of anticipating what the counterpart will actually do. Prompting exposes this limitation rather than resolving it. Training explicit theory of mind. We next train these capabilities directly through supervised distillation. GPT-5.2 generates demonstrations containing both the ordinary negotiation action and the explicit Infer and Anticipate reasoning. We refer to SFT on these complete traces as ExpToM SFT. The experiment uses no reinforcement learning, allowing us to isolate the effect of explicit reasoning supervision from outcome optimization. We compare ExpToM SFT with standard SFT on action trajectories, under both per-environment training and joint training across the four negotiation environments. Explicit ToM supervision substantially improves the modelās measured reasoning capabilities. Next-action prediction improves by roughly 3030 percentage points, while preference-inference accuracy rises to approximately 0.630.63ā0.710.71, depending on the training environment. More importantly, these gains translate into stronger negotiation. Per-environment ExpToM SFT reaches an average utility of 0.5460.546, compared with 0.5000.500 for ordinary SFT and 0.4540.454 for the base model. The improvement is consistent across all four environments. The effect also survives multi-domain training: mixed ExpToM reaches 0.5250.525, compared with 0.4950.495 for mixed ordinary SFT. Normal SFT ExpToM SFT Environment Base 4B +ToM prompt per-env mixed per-env mixed GPT-5.2 Deal-or-No-Deal 0.589 0.575 0.580 0.587 0.596 0.608 0.655 CaSiNo 0.454 0.391 0.492 0.480 0.512 0.498 0.475 Craigslist 0.295 0.225 0.445 0.409 0.552 0.478 0.511 Job Interview 0.476 0.221 0.482 0.503 0.522 0.517 0.626 Average 0.454 0.353 0.500 0.495 0.546 0.525 0.567 Table 8: Explicit theory-of-mind supervision improves negotiation. Normalized utility for the base model, prompt-only ToM, ordinary SFT, and explicit ToM SFT. ExpToM distills Inferā ā traces from GPT-5.2. Prompting the scaffold alone hurts the base 4B, while training on the same reasoning structure improves over ordinary SFT under both per-environment and mixed training. Which theory-of-mind capability matters for negotiation? Finally, we ask which component of ToM is associated with successful negotiation. Across trained checkpoints, next-action prediction accuracy is positively correlated with negotiation utility, while preference-inference accuracy shows little corresponding relationship. The distinction is visible already in the base model: it has relatively strong preference inference despite weak negotiation performance, while its largest ToM deficit is next-action prediction. This suggests that identifying the counterpartās preferences is only one part of effective social reasoning. Strategic action additionally requires anticipating how those preferences translate into behavior in response to a particular move. Explicit ToM supervision improves both capabilities, but the improvement in Anticipate is the component most closely associated with better negotiation outcomes. Taken together, these experiments provide a second route for improving social reasoning alongside outcome-based RL. Reinforcement learning trains strategic behavior from the quality of the final agreement; explicit ToM distillation instead supervises intermediate reasoning about the counterpart. The latter improves negotiation even under pure SFT, showing that part of the capability learned implicitly through interaction can also be transferred directly through structured reasoning supervision. 6 Qualitative Analysis The aggregate results in Ā§5 show that post-training substantially improves negotiation utility. We next inspect the interaction trajectories to understand how the learned policies differ from the base model. Our current analysis covers the six negotiation environments: for each environment, we compare the base model with the final domain-trained policy; where available, we also examine an SFT-only checkpoint as a diagnostic for separating behaviors introduced by imitation from those selected by reinforcement learning. We organize the analysis around two questions. First, how does training change the agentās strategic decisions: where it anchors, what it concedes, when it rejects, and what value it preserves for its principal? Second, how does training change the interaction itself: communication style and action choice? 6.1 Strategic Behavior: More Selective Concession Across the four negotiation environments, the most consistent strategic change is greater selectivity in concession. The base model often moves quickly toward agreement after encountering resistance, even when doing so gives away substantial utility. Post-training makes the policy more willing to maintain a favorable position, reject an unfavorable proposal, and concede on dimensions that are relatively inexpensive. The concrete behavior differs with the structure of each environment. Craigslist: anchor low and concede gradually. Craigslist gives the clearest view of how training reshapes the bargaining trajectory. We normalize buyer offers by the listing-to-target gap, with 00 denoting the buyerās target price, 11 the listing price, and negative values offers below the target. Table 9 shows the resulting concession curves. The base model opens essentially at its own target (+0.04+0.04), with only 3%3\% of openings below target, and then moves rapidly toward the seller. SFT introduces the missing anchoring behavior: the mean opening moves to ā1.21-1.21, and 71%71\% of games begin below the buyerās target. PPO strengthens this strategy further. The SFT++PPO policy opens at ā1.48-1.48, remains below target through its third proposal on average, and raises the below-target opening rate to 78%78\%. Stage Open 2nd 3rd 4th % open << target Untrained +0.04+0.04 +0.35+0.35 +0.41+0.41 +0.45+0.45 3% SFT ā1.21-1.21 ā0.30-0.30 ā0.08-0.08 ā0.06-0.06 71% SFT++PPO ā1.48-1.48 ā0.73-0.73 ā0.22-0.22 +0.01+0.01 78% Table 9: Craigslist buyer concession curves. Offer price is normalized by the listing-target gap: 00 denotes the buyerās target, 11 the listing price, and negative values offers below target. SFT introduces below-target anchoring; PPO produces a lower initial anchor and a slower subsequent concession trajectory. Matched trajectories illustrate the difference. With a $275 listing and a $209 target, the base model opens at $209 and eventually accepts $275, obtaining zero utility. The trained policy instead negotiates $150 ā $165, rejects $250, and eventually accepts $190, obtaining maximal utility. Training therefore changes both the initial reference point and the willingness to maintain it under pressure. Craigslist also reveals a division of labor between SFT and PPO in the language used to support these offers. The base buyer frequently reveals its own target price: 52.7%52.7\% of its messages explicitly mention that private reference point, while only 3.7%3.7\% invoke external market comparisons. After SFT, target leakage falls to 1.0%1.0\% and market/comparable references rise to 29.6%29.6\% of messages. PPO subsequently shifts toward commitment language: explicit āfinal offer,ā āfirm limit,ā and related expressions rise to 13.3%13.3\% of messages. Thus, SFT introduces useful bargaining moves and rhetorical forms, while PPO selects when to commit to them. One undesirable behavior also appears here: the environment provides no external comparable-price information, so some learned market references are fabricated. We treat this learned bluffing behavior as a limitation rather than as evidence of improved factual grounding. Deal-or-No-Deal: resist capitulation after rejection. In Deal-or-No-Deal, the main weakness of the base model appears after its initial proposal. Its openings are often reasonable, but rejection can trigger a sharp concession or repeated proposals that eventually lead to failure. For example, in one trajectory where hats and balls are each worth 55 and books are worth 00, the base model responds to resistance by accepting only a book, giving away all of its value. Other trajectories repeat nearly identical allocations until the counterpart walks away. PPO makes rejection less likely to trigger this collapse. The trained policy more consistently preserves high-value items and either counters or accepts when the available allocation is favorable. Relative to the SFT diagnostic checkpoint, PPO also reduces the zero-reward and no-deal tail: zero-reward episodes fall from 4.7%4.7\% to 2.5%2.5\%, and no-deals from 4.5%4.5\% to 0.5%0.5\%. The qualitative shift is therefore concentrated in how the policy responds when its preferred allocation is challenged. CaSiNo: maintain a floor on high-priority resources. CaSiNo exhibits a closely related pattern. Here, successful negotiation requires protecting high-priority resources while using low-priority resources as concessions. The base model often concedes too much after encountering resistance. In a matched example, it initially asks for all three food packages and, after one rejection, immediately drops to a single food package. PPO learns a more stable reservation strategy. The trained policy explicitly maintains floors such as āI require at least 2 food packages,ā preserves its high- and medium-priority resources, and gives ground primarily on its low-priority resource. At the distribution level, the fraction of over-conceding games falls from 17.3%17.3\% to 13.6%13.6\%, while the fraction of games in which the agent strongly preserves its own value rises from 17.3%17.3\% to 24.2%24.2\%. The SFT diagnostic moves in the opposite direction, raising over-concession to 29.4%29.4\%, suggesting that imitation alone can favor agreeable behavior without calibrating how much value to surrender. Job Interview: protect high-weight contract terms. Job Interview makes selective concession multi-dimensional. The two parties negotiate salary and holiday together with categorical terms such as position, workplace, and company. A strong policy therefore needs to identify which dimensions carry the largest private utility and spend concessions on less important issues. The trained trajectories show greater preservation of high-value categorical terms. The workerās categorical utility in completed agreements rises from approximately 0.640.64 for the base model to 0.730.73ā0.780.78 for the trained checkpoints, while the fraction of agreements securing the workerās top-choice workplace increases from 39%39\% to 6868ā72%72\%. In matched examples, the base model can abandon several high-value categorical terms in order to close quickly, whereas the trained policy holds position, workplace, and company fixed while negotiating over salary and holiday. Marketplace: create bargaining room and protect the reservation price. Marketplace exhibits a particularly sharp strategic shift. The base buyer is already capable of completing transactions, but negotiates from an extremely weak position: its mean opening offer is 0.9560.956 of its private reservation price, leaving almost no room to bargain. After PPO, the mean opening falls to 0.3610.361 of the reservation price. The trained policy therefore begins far from its maximum willingness to pay and preserves substantial concession room. Training also eliminates a severe information-leakage failure mode. In the base policy, 62%62\% of opening messages reveal the buyerās reservation price or explicitly state its budget constraint. After PPO, this behavior essentially disappears. The learned shift closely parallels Craigslist: the agent stops revealing its private boundary, anchors aggressively, and moves toward agreement gradually rather than beginning near its own limit. Calendar: optimize agreement quality rather than agreement rate. Calendar exposes a different failure mode. The central challenge is not anchoring or bargaining over a scalar surplus, but selecting a mutually feasible time slot that also serves the principalās private preferences. Before Calendar training, the policy is highly willing to schedule a meeting but frequently accepts the counterpartās proposed slot without comparing it against better alternatives. Using the Marketplace-trained checkpoint as a pre-Calendar proxy, 92%92\% of games end in a scheduled meeting, yet only 16%16\% obtain a maximum-preference slot and 63%63\% receive zero utility. Calendar PPO changes this objective. The trained policy schedules fewer meetings overall (74%74\%), but the quality of the selected slots improves substantially: the fraction of maximum-utility outcomes rises from 16%16\% to 38%38\%, while zero-utility outcomes fall from 63%63\% to 35%35\%. The policy has therefore learned that completing the coordination task is insufficient; it should reject or counter-propose when the available agreement poorly serves its principal. Cross-domain pattern. Across all six environments, training makes agreement more explicitly conditional on the principalās utility. The base model often treats reaching an agreement as valuable in itself and gives ground too readily when the counterpart resists or proposes a feasible alternative. Post-training instead learns environment-specific reservation behavior: Craigslist and Marketplace create bargaining room through aggressive anchors and slower price concessions; DnD resists capitulation after rejection; CaSiNo protects high-priority resources; Job Interview preserves high-weight contract terms; and Calendar rejects feasible but low-value slots in favor of higher-preference alternatives. The common strategic change is therefore utility-sensitive concession: the policy becomes better at distinguishing what can be traded away from what should be protected. The particular implementation depends on the interaction structure: a reservation price in bilateral bargaining, a preferred bundle in allocation, high-weight issues in multi-issue negotiation, or a preference-weighted time slot in coordination. 6.2 Interaction Behavior: Less Private Deliberation, More Strategic Communication The strategic changes above are accompanied by a broad shift in how the policy conducts the interaction. Table 10 summarizes several trajectory-level measures. Across the four analyzed environments, trained policies generate substantially fewer total tokens per turn while sending longer and more informative messages to the counterpart. They also use rejection more actively and exhibit more explicit commitment language. Env Stage Turns Polite Firm CaSiNo untrained 2.5 16% 1% SFT 3.4 10% 4% PPO 3.4 59% 11% DnD untrained 2.9 8% 2% SFT 3.1 15% 23% PPO 3.2 36% 2% Craigslist untrained 4.1 53% 3% SFT 5.4 15% 3% SFT++PPO 4.6 51% 23% Job Interview untrained 4.6 35% 2% SFT 5.5 12% 7% SFT++PPO 3.8 40% 43% Table 10: Interaction behavior across the training ladder. Turns denotes turns per game; public tok is the token length of the opponent-facing message; total tok includes the full generation associated with each turn; <think> is the fraction of turns containing an explicit reasoning block; polite and firm denote the fractions of public messages containing courteous and commitment language, respectively. Tokens are measured with the Qwen3-4B tokenizer. Rejection becomes a strategic action. The base policies rely heavily on proposing and accepting, with explicit rejection appearing in only 00ā3%3\% of actions in the analyzed traces. Training makes rejection a meaningful part of the policy. Reject actions rise to approximately 14%14\% in DnD and 20%20\% in Job Interview, and they are typically accompanied by a counterproposal or explanation of which terms are unacceptable. This change complements the selective-concession pattern above. The trained model has acquired a practical way to maintain a reservation boundary: instead of responding to an unfavorable proposal by immediately moving toward it, the policy can reject, explain the conflict, and propose an alternative. Commitment becomes firmer while remaining socially cooperative. Training also changes the linguistic form of bargaining. Commitment language increases substantially in several environments: from 1%1\% to 11%11\% of messages in CaSiNo, 3%3\% to 23%23\% in Craigslist, and 2%2\% to 43%43\% in the analyzed Job Interview checkpoint. These messages contain explicit boundaries such as āfirm,ā āfinal offer,ā or ānon-negotiable.ā Firmness generally coexists with courteous language. For example, politeness rises from 16%16\% to 59%59\% in CaSiNo and remains around half of Craigslist messages in the final policy. The learned behavior is therefore closer to polite commitment: clear reservation boundaries expressed through cooperative language. SFT broadens behavior; RL selects behavior by utility. The intermediate SFT checkpoints provide a useful diagnostic of how the two training signals differ. In Craigslist, SFT introduces capabilities that are almost absent from the base model: below-target anchoring, external justifications, and reduced leakage of the buyerās private target. RL then changes how these behaviors are deployed, producing stronger commitment and slower concession. In the other environments where we evaluate SFT as an ablation, imitation can also introduce undesirable behavior: more looping in DnD, more over-concession in CaSiNo, and weaker categorical preservation in the currently analyzed Job Interview traces. Outcome optimization subsequently selects among these behaviors according to realized utility. This provides a trajectory-level interpretation of the training recipe: supervised data can expand the policyās behavioral repertoire, while reinforcement learning determines which parts of that repertoire are strategically useful. 7 Conclusion We introduced SocialRL, a framework for training and studying language-model agents in heterogeneous delegated interactions. At the systems level, we develop a general event-based multi-agent environment interface that is independent of agent implementation, together with a decoupled training infrastructure that separates rollout generation from optimization through an OpenAI-compatible rollout proxy. These abstractions allow the same environments and agent harnesses to support local or remote agents, heterogeneous and black-box counterparts, reinforcement learning, and distillation without coupling the interaction stack to a particular trainer. On top of this infrastructure, we study social-reasoning post-training across six heterogeneous negotiation and coordination environments. Domain-specific training produces strong 4B specialists whose aggregate performance lies in the range of much larger GPT models. Evaluating these specialists across all environments reveals substantial and highly directional transfer: structurally related interactions reinforce one another, while other training configurations introduce interference. We exploit this structure in two complementary approaches to unification. Transfer-aware cascade RL produces a single 4B policy with an Avg-6 of 0.6270.627, comparable to GPT-4.1, GPT-5.1, and GPT-5.2, and substantially outperforms random and anti-transfer curricula. MOPD provides a more efficient alternative, recovering most of the specialistsā advantage in fewer than 100100 additional optimization steps. We further investigate explicit theory-of-mind supervision as a complementary source of social-reasoning capability. Distilling Inferā ā traces improves both negotiation performance and measured theory-of-mind abilities, with next-action prediction emerging as the component most closely associated with negotiation outcomes. Trajectory analysis provides a consistent behavioral interpretation of these gains: trained agents become more sensitive to their principalās utility, protect private information more effectively, concede more selectively, and adapt their reasoning strategy to the structure of the interaction. Taken together, the environment abstraction, decoupled training infrastructure, and training results provide a general platform for studying social reasoning in interactive agents. Our results suggest that strategic capabilities learned in one interaction structure can transfer to others, that this transfer can be exploited when consolidating specialists, and that strong multi-domain social behavior can be learned in relatively small models. References Allouah et al. (2026) A. Allouah, O. Besbes, J. D. Figueroa, Y. Kanoria, and A. Kumar What is your ai agent buying? evaluation, biases, model dependence, & emerging implications of agentic e-commerce. In Proceedings of the ACM Web Conference 2026, p. 8697ā8700. Cited by: Ā§2. Anantaprayoon et al. (2026) P. Anantaprayoon, N. Babina, N. Asgharbeygi, and J. Tarifi Learning to negotiate: multi-agent deliberation for collective value alignment in llms. arXiv preprint arXiv:2603.10476. Cited by: Ā§2, Ā§2. Awadallah et al. (2025) A. Awadallah, Y. Lara, R. Magazine, H. Mozannar, A. Nambi, Y. Pandya, A. Rajeswaran, C. Rosset, A. Taymanov, V. Vineet, et al. Fara-7b: an efficient agentic model for computer use. arXiv preprint arXiv:2511.19663. Cited by: Ā§2. Bergemann et al. (2026) D. Bergemann, S. Ghili, X. Hu, C. Li, and Z. Yang Training language models for bilateral trade with private information. arXiv preprint arXiv:2604.16472. Cited by: Ā§2. Bianchi et al. (2024) F. Bianchi, P. J. Chia, M. Yuksekgonul, J. Tagliabue, D. Jurafsky, and J. Zou How well can llms negotiate? negotiationarena platform and analysis. arXiv preprint arXiv:2402.05863. Cited by: Ā§1, Ā§2. Chaturvedi and Verma (2023) R. Chaturvedi and S. Verma Opportunities and challenges of ai-driven customer service. Artificial Intelligence in customer service: The next frontier for personalized engagement, p. 33ā71. Cited by: Ā§1. Chawla et al. (2021) K. Chawla, J. Ramirez, R. Clever, G. Lucas, J. May, and J. Gratch Casino: a corpus of campsite negotiation dialogues for automatic negotiation systems. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, p. 3167ā3185. Cited by: Ā§1, Ā§1, Ā§4.1. Dahlgren Lindstrƶm et al. (2025) A. Dahlgren Lindstrƶm, L. Methnani, L. Krause, P. Ericson, Ć. M. de Rituerto de Troya, D. Coelho Mollo, and R. Dobbe Helpful, harmless, honest? sociotechnical limits of ai alignment and safety through reinforcement learning from human feedback: ad lindstrƶm et al.. Ethics and Information Technology 27 (2), p. 28. Cited by: Ā§1. Dong et al. (2024) H. Dong, W. Xiong, B. Pang, H. Wang, H. Zhao, Y. Zhou, N. Jiang, D. Sahoo, C. Xiong, and T. Zhang Rlhf workflow: from reward modeling to online rlhf. arXiv preprint arXiv:2405.07863. Cited by: Ā§1. Du et al. (2024) Y. Du, R. Srikant, and W. Chen Cascading reinforcement learning. In International Conference on Learning Representations, Vol. 2024, p. 30263ā30304. Cited by: Ā§1, Ā§5.3. Elkoussy and Perez (2026) L. Elkoussy and J. Perez AgentLTL: a trace-verification framework for measuring, enforcing, and training procedural compliance in tool-using llm agents. arXiv preprint arXiv:2607.02599. Cited by: Ā§2. Fu et al. (2026) S. Fu, H. Xu, R. He, Z. Fu, H. Wang, J. Huang, X. Ma, M. Zhong, W. Huang, X. He, et al. Poly-opd: heterogeneous multi-teacher on-policy distillation for capability-selectable flow models. arXiv preprint arXiv:2608.04349. Cited by: Ā§5.3. Fu et al. (2023) Y. Fu, H. Peng, T. Khot, and M. Lapata Improving language model negotiation with self-play and in-context learning from ai feedback. arXiv preprint arXiv:2305.10142. Cited by: Ā§2. Google (2026) Google Google scheduler. https://workspace.google.com/resources/appointment-scheduling/. Cited by: Ā§1. Haurum et al. (2024) K. R. Haurum, R. Ma, and W. Long Real estate with ai: an agent based on langchain. Procedia Computer Science 242, p. 1082ā1088. Cited by: Ā§1. He et al. (2018) H. He, D. Chen, A. Balakrishnan, and P. Liang Decoupling strategy and generation in negotiation dialogues. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, p. 2333ā2343. Cited by: Ā§1, Ā§1, Ā§4.1. Hua et al. (2024) W. Hua, O. Liu, L. Li, A. Amayuelas, J. Chen, L. Jiang, M. Jin, L. Fan, F. Sun, W. Wang, et al. Game-theoretic llm: agent workflow for negotiation games. arXiv preprint arXiv:2411.05990. Cited by: Ā§1. Hwang et al. (2026) E. Hwang, Y. Yin, G. Carenini, P. West, and V. Shwartz Infusing theory of mind into socially intelligent llm agents. In Findings of the Association for Computational Linguistics: ACL 2026, p. 11327ā11360. Cited by: Ā§5.4. Jiang et al. (2026) B. Jiang, T. Shi, R. Kamoi, Y. Yuan, C. J. Taylor, L. Yang, P. Zhou, and S. Chen One model, all roles: multi-turn, multi-agent self-play reinforcement learning for conversational social intelligence. arXiv preprint arXiv:2602.03109. Cited by: Ā§2. Juneja et al. (2025) G. Juneja, J. Pasupulati, A. Albalak, W. Hua, and W. Y. Wang MAGPIE: a benchmark for multi-agent contextual privacy evaluation. In Socially Responsible and Trustworthy Foundation Models at NeurIPS 2025, Cited by: Ā§1. Kirk et al. (2024) R. Kirk, I. Mediratta, C. Nalmpantis, J. Luketina, E. Hambro, E. Grefenstette, and R. Raileanu Understanding the effects of rlhf on llm generalisation and diversity. In International Conference on Learning Representations, Vol. 2024, p. 20620ā20653. Cited by: Ā§1. Kostka and Chudziak (2025) A. Kostka and J. A. Chudziak Evaluating theory of mind and internal beliefs in llm-based multi-agent systems. In International Conference on Computational Collective Intelligence, p. 18ā32. Cited by: Ā§5.4. Kovala (2026) T. Kovala A complete guide to agentforce. Springer Books. Cited by: Ā§1. Lewis et al. (2017) M. Lewis, D. Yarats, Y. Dauphin, D. Parikh, and D. Batra Deal or no deal? end-to-end learning of negotiation dialogues. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, p. 2443ā2453. Cited by: Ā§1, Ā§1, Ā§4.1. Liao et al. (2024) A. Liao, N. Tomlin, and D. Klein Efficacy of language model self-play in non-zero-sum games. arXiv preprint arXiv:2406.18872. Cited by: Ā§2. Liu et al. (2026) S. D. Liu, C. Chen, J. S. Xiao, L. Lei, Y. Zhang, Y. Yue, and D. Simchi-Levi Instructing llms to negotiate using reinforcement learning with verifiable rewards. arXiv preprint arXiv:2604.09855. Cited by: Ā§2. Luo et al. (2025) X. Luo, Y. Zhang, Z. He, Z. Wang, S. Zhao, D. Li, L. K. Qiu, and Y. Yang Agent lightning: train any ai agents with reinforcement learning. arXiv preprint arXiv:2508.03680. Cited by: Ā§1. Lyu et al. (2026) N. Lyu, B. Xu, X. Tian, W. Meng, Y. Yuan, Y. Zhang, Z. Huang, T. Chua, and H. Shen GIFT: games as informal training for generalizable llms. arXiv preprint arXiv:2601.05633. Cited by: Ā§2. Ma et al. (2026) W. Ma, J. Wei, L. Zhao, H. Zhang, B. Xiao, L. Li, Q. Yang, B. Gao, Y. Wang, R. Li, et al. Mopd: multi-teacher on-policy distillation for capability integration in llm post-training. arXiv preprint arXiv:2606.30406. Cited by: Ā§5.3. Matta (2026) D. Matta Artificial intelligence and theory of mind. Journal of Psychology and AI 2 (1), p. 2628373. Cited by: Ā§1. Microsoft Research, AI Frontiers (2026) Microsoft Research, AI Frontiers SocialReasoning-bench: measuring whether ai agents act in usersā best interests. Note: https://w.microsoft.com/en-us/research/blog/socialreasoning-bench-measuring-whether-ai-agents-act-in-users-best-interests/ Cited by: Ā§1, Ā§4.1, Ā§4.1, Ā§4.2, Ā§4.2. Mu et al. (2026) C. Mu, Y. Zeng, Q. Zhang, K. Shao, C. Chu, H. Guo, D. Jia, Z. Wang, and S. Hu Adaptive theory of mind for llm-based multi-agent coordination. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, p. 29608ā29616. Cited by: Ā§5.4. OpenAI (2025) OpenAI Operator system card.. https://openai.com/index/operator-system-card. Cited by: Ā§1. Schick et al. (2023) T. Schick, J. Dwivedi-Yu, R. DessƬ, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom Toolformer: language models can teach themselves to use tools. Advances in neural information processing systems 36, p. 68539ā68551. Cited by: Ā§2. South et al. (2025) T. South, S. Marro, T. Hardjono, R. Mahari, C. D. Whitney, D. Greenwood, A. Chan, and A. Pentland Authenticated delegation and authorized ai agents. arXiv preprint arXiv:2501.09674. Cited by: Ā§1. Sun et al. (2025a) C. Sun, S. Huang, and D. Pompili Llm-based multi-agent decision-making: challenges and future directions. IEEE Robotics and Automation Letters 10 (6), p. 5681ā5688. Cited by: Ā§1. Sun et al. (2025b) H. Sun, Y. Wu, Y. Cheng, and X. Chu Game theory meets large language models: a systematic survey. arXiv preprint arXiv:2502.09053. Cited by: Ā§1. Wang et al. (2025) B. Wang, C. Lee, N. Lee, S. Lin, W. Dai, Y. Chen, Y. Chen, Z. Yang, Z. Liu, M. Shoeybi, et al. Nemotron-cascade: scaling cascaded reinforcement learning for general-purpose reasoning models. arXiv preprint arXiv:2512.13607. Cited by: Ā§5.3. Wang et al. (2024) R. Wang, H. Yu, W. Zhang, Z. Qi, M. Sap, Y. Bisk, G. Neubig, and H. Zhu Sotopia-Ļ: interactive learning of socially intelligent language agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 12912ā12940. Cited by: Ā§2. Wang et al. (2026a) Y. Wang, X. Chen, X. Jin, M. Wang, and L. Yang Openclaw-rl: train any agent simply by talking. arXiv preprint arXiv:2603.10165. Cited by: Ā§1. Wang et al. (2026b) Z. Wang, C. Xu, B. Liu, Y. Wang, S. Han, Z. Yao, H. Yao, and Y. He Agent world model: infinity synthetic environments for agentic reinforcement learning. arXiv preprint arXiv:2602.10090. Cited by: Ā§2. Xiao et al. (2025) Y. Xiao, J. Wang, Q. Xu, C. Song, C. Xu, Y. Cheng, W. Li, and P. Liu Towards dynamic theory of mind: evaluating llm adaptation to temporal evolution of human states. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 24036ā24057. Cited by: Ā§5.4. Xie et al. (2024) T. Xie, D. Zhang, J. Chen, X. Li, S. Zhao, R. Cao, T. J. Hua, Z. Cheng, D. Shin, F. Lei, et al. Osworld: benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems 37, p. 52040ā52094. Cited by: Ā§2. Yamaguchi et al. (2021) A. Yamaguchi, K. Iwasa, and K. Fujita Dialogue act-based breakdown detection in negotiation dialogues. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, p. 745ā757. Cited by: Ā§1, Ā§4.1. Yang and Zhang (2024) J. Yang and J. Zhang A multi-teacher policy distillation framework for enhancing zero-shot generalization of autonomous driving policies. IEEE Transactions on Vehicular Technology 73 (7), p. 9734ā9746. Cited by: Ā§5.3. Yang et al. (2026a) S. Yang, C. Han, X. Ma, Y. Li, M. R. G. Madani, and E. Hovy Evotool: self-evolving tool-use policy optimization in llm agents via blame-aware mutation and diversity-aware selection. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 43553ā43572. Cited by: Ā§2. Yang et al. (2026b) W. Yang, W. Liu, R. Xie, K. Yang, S. Yang, and Y. Lin Learning beyond teacher: generalized on-policy distillation with reward extrapolation. arXiv preprint arXiv:2602.12125. Cited by: Ā§5.3. Yao et al. (2024) S. Yao, N. Shinn, P. Razavi, and K. Narasimhan Tau-bench: a benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.12045. Cited by: Ā§2, Ā§2. Yao et al. (2022) S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao React: synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629. Cited by: Ā§2. Yu et al. (2025) H. Yu, Z. Qi, Y. Zhao, K. Nottingham, K. Xuan, B. P. Majumder, H. Zhu, P. P. Liang, and J. You Sotopia-rl: reward design for social intelligence. arXiv preprint arXiv:2508.03905. Cited by: Ā§2. Yuan et al. (2025) H. Yuan, Z. Xu, Z. Tan, X. Yi, M. Guang, K. Long, H. Hui, B. Li, X. Chen, B. Zhao, et al. MARS: reinforcing multi-agent reasoning of llms through self-play in strategic games. arXiv e-prints, p. arXivā2510. Cited by: Ā§2. Zeng et al. (2025) A. Zeng, X. Lv, Q. Zheng, Z. Hou, B. Chen, C. Xie, C. Wang, D. Yin, H. Zeng, J. Zhang, et al. Glm-4.5: agentic, reasoning, and coding (arc) foundation models. arXiv preprint arXiv:2508.06471. Cited by: Ā§1. Zhang et al. (2026) E. Zhang, F. Zhang, A. Pappu, B. El, J. Blanchet, S. Athey, J. Liu, and J. Zou TERMS-bench: diagnosing llm negotiation agents beyond deal rate. arXiv preprint arXiv:2605.13909. Cited by: Ā§1. Zhou et al. (2023) J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. Hou Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911. Cited by: Ā§1. Zhou et al. (2024) S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y. Bisk, D. Fried, et al. Webarena: a realistic web environment for building autonomous agents. In International Conference on Learning Representations, Vol. 2024, p. 15585ā15606. Cited by: Ā§2. Zhu et al. (2026) C. Zhu, S. Cui, W. S. Dufallo, R. Jin, Z. Xu, L. Zhang, and D. Cain PieArena: ranking and profiling language agents in realistic negotiation scenarios. arXiv preprint arXiv:2602.05302. Cited by: Ā§2. Zhu et al. (2025) S. Zhu, J. Sun, Y. Nian, T. South, A. Pentland, and J. Pei The automated but risky game: modeling and benchmarking agent-to-agent negotiations and transactions in consumer markets. arXiv preprint arXiv:2506.00073. Cited by: Ā§1. Zillow (2026) Zillow Zillow debuts ai mode. https://w.zillow.com/news/zillow-debuts-ai-mode. Cited by: Ā§1. Zou et al. (2026) C. Zou, Y. Yao, S. She, N. Goodman, and R. D. Hawkins CalBench: evaluating coordination-privacy trade-offs in multi-agent llms. arXiv preprint arXiv:2605.09823. Cited by: Ā§1, Ā§2.