Paper deep dive

One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RL

Simon Yu, Nicholas Tomlin, Marwa Abdulhai, Ximing Lu, Derek Chong, Abe Hou, Dilara Soylu, Sergey Levine, Christopher D. Manning, Weiyan Shi

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 149

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%

Last extracted: 8/18/2026, 4:26:30 AM

Summary

This paper identifies 'simulator collapse' as a critical failure mode in multi-agent reinforcement learning (MARL) for human-AI interaction, where training an LLM policy against a single frozen LLM simulator leads to overfitting on narrow, exploitative strategies due to the simulator's mode-collapsed behavior. The authors propose two solutions: Verbalized Sampling (inference-time) to broaden simulator diversity, and Co-Training (training-time) to jointly optimize the policy and a trainable simulator population. Validated on Persuasion for Good, τ²-bench, and CooperBench, these methods significantly improve generalization to unseen simulators and real users. The paper also introduces SCOPE, an open-source framework for Population Co-Training.

Entities (9)

Simulator Collapse · concept · 98%Co-training · method · 97%Verbalized Sampling · method · 97%SCOPE · softwareframework · 96%CooperBench · benchmark · 95%Persuasion for Good · benchmark · 95%τ-bench · benchmark · 95%Multi-Agent Reinforcement Learning · field · 94%mode collapse · phenomenon · 93%

Relation Signals (7)

Single-simulator RL → suffersfrom → Simulator Collapse

confidence 97% · Single-simulator RL collapses; Verbalized Sampling and Co-Training recover.

Verbalized Sampling → mitigates → Simulator Collapse

confidence 96% · Verbalized Sampling, broadens the simulator's behavior by sampling from a verbalized response distribution, reducing mode collapse.

Co-training → mitigates → Simulator Collapse

confidence 96% · Co-Training, jointly optimizes the policy against a population of trainable simulators, preventing it from overfitting to any single simulator's mode.

Simulator Collapse → causes → Policy Overfitting

confidence 95% · an LLM policy trained against it overfits to narrow strategies that exploit the simulator's dominant mode

SCOPE → implements → Co-training

confidence 94% · we release SCOPE, an open-source framework for Population Co-Training multi-agent RL.

Co-training → improves → Held-out Success

confidence 93% · Co-Training pushes gains further to 14%

Verbalized Sampling → improves → Held-out Success

confidence 93% · Verbalized Sampling improves held-out success by up to 9% over single-simulator RL

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Multi-agent reinforcement learning for human-AI interaction typically relies on a single large language model to simulate user behavior. We show that this approach systematically fails to generalize, and trace the failure to simulator collapse: because the simulator LLM is mode-collapsed, an LLM policy trained against it overfits to narrow strategies that exploit the simulator's dominant mode, and such a policy transfers poorly to unseen simulators and real users. We formalize this collapse theoretically and propose two complementary solutions, one at inference time and one at training time. The inference-time solution, Verbalized Sampling, broadens the simulator's behavior by sampling from a verbalized response distribution, reducing mode collapse. The training-time solution, Co-Training, jointly optimizes the policy against a population of trainable simulators, preventing it from overfitting to any single simulator's mode. We validate both solutions on three multi-turn benchmarks: Persuasion for Good, $\tau^2$-bench, and CooperBench. Verbalized Sampling improves held-out success by up to 9% over single-simulator RL, and Co-Training pushes gains further to 14%; the human study shows similar gain on real users. Both solutions preserve the policy diversity that collapses under single-simulator RL. To support further work in this direction, we release SCOPE, an open-source framework for Population Co-Training multi-agent RL. More broadly, our results suggest that the diversity of the training environment, not only the policy, is critical to the generalization of multi-turn RL to real-world deployment.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

145,927 characters extracted from source content.

Expand or collapse full text

One Frozen Simulator Is Not Enough: Simulator Collapse in Multi-Agent RL Simon Yu Affiliation: Northeastern University Nicholas Tomlin Affiliation: New York University Marwa Abdulhai Affiliation: UC Berkeley Ximing Lu Affiliation: University of Washington Derek Chong Affiliation: Stanford University Abe Hou Affiliation: Stanford University Dilara Soylu Affiliation: Stanford University Sergey Levine Affiliation: UC Berkeley Christopher D. Manning Affiliation: Stanford University Weiyan Shi Affiliation: Northeastern University Abstract Multi-agent reinforcement learning for human-AI interaction typically relies on a single large language model to simulate user behavior. We show that this approach systematically fails to generalize, and trace the failure to simulator collapse: because the simulator LLM is mode-collapsed, an LLM policy trained against it overfits to narrow strategies that exploit the simulator’s dominant mode, and such a policy transfers poorly to unseen simulators and real users. We formalize this collapse theoretically and propose two complementary solutions, one at inference time and one at training time. The inference-time solution, Verbalized Sampling, broadens the simulator’s behavior by sampling from a verbalized response distribution, reducing mode collapse. The training-time solution, Co-Training, jointly optimizes the policy against a population of trainable simulators, preventing it from overfitting to any single simulator’s mode. We validate both solutions on three multi-turn benchmarks: Persuasion for Good, τ2τ^2-bench, and CooperBench. Verbalized Sampling improves held-out success by up to 9% over single-simulator RL, and Co-Training pushes gains further to 14%; the human study shows similar gain on real users. Both solutions preserve the policy diversity that collapses under single-simulator RL. To support further work in this direction, we release SCOPE, an open-source framework for Population Co-Training multi-agent RL. More broadly, our results suggest that the diversity of the training environment, not only the policy, is critical to the generalization of RL to real-world deployment11 1 We released the code at https://github.com/CHATS-lab/scope_usim. Figure 1: Single-simulator RL collapses; Verbalized Sampling and Co-Training recover. τ2τ^2-bench (Qwen3-4B-Instruct). (a) Held-out generalization: RL against a single frozen simulator peaks early, then starts collapsing. (b) Policy entropy: the same recipe drives the policy’s entropy to near zero. (c) Human study: both fixes lift real-user performance over RL (Single), which drops below even the untrained baseline. 1 Introduction Reinforcement learning with verifiable rewards has driven rapid progress on single-turn LLM capabilities, from mathematical reasoning (DeepSeek-AI 2025; Liu et al. 2025a; Yu et al. 2025a) to tool use (Feng et al. 2025; Jin et al. 2025) to software engineering (Yang et al. 2025a; Wei et al. 2025). These settings share a loop: generate an answer, verify it against the environment, and update the policy. Toward more realistic tasks beyond single-agent verifiable rewards, a growing line of work studies multi-turn human-AI interaction (Qian et al. 2025; Zhou et al. 2025a; Wu et al. 2026): customer support (Barres et al. 2025), collaborative coding (Khatua et al. 2026; Wang et al. 2026a), persuasion (Wang et al. 2019), and tutoring (Abdulhai et al. 2025). But training RL with real users at scale is prohibitively expensive and slow, so prior work has turned to LLM-based user simulators Park et al. 2023; Park et al. 2024. A common method in recent work is to prompt a single frozen LLM to play the user Qian et al. 2025; Yu et al. 2025b. In this work, we show this practice can systematically fail to generalize. In multi-turn RL (Abdulhai et al. 2023), the policy no longer interacts with a deterministic verifier; the simulated user becomes part of the training environment, and its output distribution decides which states the policy visits and what gradient signals it receives. We identify a systematic failure mode of this recipe, which we call simulator collapse. Because many aligned LLM simulators are mode-collapsed (Jiang et al. 2025; GX-Chen et al. 2025; Zhang et al. 2025a), a policy trained against a single frozen simulator receives gradients dominated by that simulator’s modal behavior, and overfits to narrow strategies that exploit that mode (MacDiarmid et al. 2025). The error compounds across dialogue turns, so a policy trained this way fails when transferred to unseen simulators, and is unlikely to transfer to real users. First, in Section 3, we formalize simulator collapse. A mode-collapsed simulator does not necessarily make the policy gradient vanish; it biases the gradient toward the simulator’s mode. Repeated policy updates then rank the policy’s trajectories by how well they exploit that mode, concentrating probability on a narrow exploit set. This explains the rapid policy-entropy drop we observe in training. The resulting policy performs well in-distribution but fails on unseen simulators whose responses contain behaviors the training simulator didn’t produce. Two solutions follow from the theory, at different phases of the training loop: inference-time and training-time (Section 3.4). First, Verbalized Sampling (Zhang et al. 2025a) is the inference-time solution. At each simulator turn during rollout, the simulator is queried for a verbalized response distribution and a response is sampled from it, restoring within-simulator diversity without retraining. Second, Co-Training is the training-time solution. We update the user simulator alongside the policy on the same conversation (Liu et al. 2025b; Liu et al. 2025c). The two sides then co-evolve: the simulator’s mode at each history shifts as the policy improves, so the strategy that exploited the existing mode no longer wins against the future one. The policy therefore faces a partner that is evolving across training. Section 4 validates both solutions across three settings: Persuasion for Good (Wang et al. 2019) (adversarial dialogue), τ2τ^2-bench (Barres et al. 2025) (collaborative tool-calling), and CooperBench (Khatua et al. 2026) (collaborative software engineering). Across all three benchmarks, Population Co-Training reaches the highest held-out task success and preserves policy entropy. The main contributions can be summarized as follows: 1. Identifying simulator collapse. We formalize how a mode-collapsed user simulator biases the policy gradient toward its mode and collapses the policy’s entropy onto a narrow simulator-specific exploit. This defines a structural failure mode of RL against a single frozen simulator (§3.2). 2. Two solutions at different points in the training loop. Verbalized Sampling (Zhang et al. 2025a) is the inference-time solution. At each simulator turn during rollout, the simulator is queried for a verbalized response distribution and one response is sampled from it, restoring within-simulator diversity without retraining either side. Co-Training is the training-time solution. We jointly update the policy and a trainable user simulator on the same rollouts, so the partner adapts as the policy improves. We release SCOPE, an open-source framework that unifies multi-model rotation, self-play, and dual-model Co-Training behind a single pluggable interface (§3.4). 3. Empirical validation across three multi-agent RL settings. On Persuasion for Good, τ2τ^2-bench, and CooperBench, single-simulator RL drops back toward the untrained baseline. Both Verbalized Sampling and Co-Training close most of the held-out gap, and Population Co-Training yields the strongest held-out task success rate (§4). We additionally run a human study on τ2τ^2-bench and Persuasion for Good, where Co-Training improves task outcome and both methods improve P4G dialogue naturalness over single-simulator RL (Appendix E). 2 Background Multi-agent RL as a POMDP. We model multi-turn dialogue as a two-player partially observable Markov decision process (POMDP) with a shared conversation-history state. Of the three settings we study, Persuasion for Good and τ2τ^2-bench are partially observable stochastic games (POSGs): the user simulator has private observations (goal, persona) the agent does not see, and only the agent receives task reward. CooperBench is the Dec-POMDP case : two cooperating coding agents share a task-success reward. We use the unified POMDP abstraction with shared history because the theory in §3.2 only depends on the joint trajectory distribution, which is invariant to how private observations are partitioned. At each turn t, the state st=(o0,a0π,a0ϕ,…,ot−1,at−1π,at−1ϕ)s_t=(o_0,a_0^π,a_0^φ,…,o_t-1,a_t-1^π,a_t-1^φ) is the full conversation history. The agent samples its utterance atπ∼πθ(⋅∣st)a_t^π _θ(· s_t); the user simulator then samples a response atϕ∼ϕψ(⋅∣st,atπ)a_t^φ _ψ(· s_t,a_t^π). A trajectory τ=(s0,a0π,a0ϕ,…,sT)τ=(s_0,a_0^π,a_0^φ,…,s_T) has terminal reward R⁡(τ)R(τ), and the agent maximizes J⁡(θ,ψ)=τ∼(πθ,ϕψ)[R⁡(τ)].J(θ;ψ)=E_τ ( _θ, _ψ)[R(τ)]. (1) The state-visitation distribution dψπθ(s)=∑tPr⁡(st=s∣πθ,ϕψ)d _θ_ψ(s)= _t (s_t=s _θ, _ψ) is jointly determined by the agent and the simulator: the simulator does not only score trajectories, it determines which histories the policy learns from. When the simulator is fixed we abbreviate Jϕ(θ)=J⁡(θ,ψ)J_φ(θ)=J(θ;ψ). Policy update. We apply REINFORCE (Williams 1992) to full multi-turn trajectories using group-relative reward normalization. For each task we sample a group of G trajectories, score them by terminal reward R⁡(τn)R(τ^n), and form z-scored advantages A^n=(R⁡(τn)−R¯)/σR A^n=(R(τ^n)- R)/ _R (Eq. 2), assigned uniformly to all agent tokens in each trajectory. If all trajectories in a group receive the same terminal reward, σR=0 _R=0 and the update stalls; this is one boundary case. More generally, σR _R can remain positive, but if the simulator keeps responding in the same way at the histories visited during training, the remaining contrast mostly ranks agent samples by how well they exploit that simulator. Section 3 formalizes this active but biased gradient. A^n=R⁡(τn)−R¯σR,R¯=1G∑n=1GR⁡(τn),σR=1G∑n=1G(R⁡(τn)−R¯)2. A^n= R(τ^n)- R _R, R= 1G _n=1^GR(τ^n), _R= 1G _n=1^G(R(τ^n)- R)^2. (2) 3 Simulator Collapse: Why One Simulator Is Not Enough Figure 2: Simulator collapse and our two fixes. (a) Problem: the real-user distribution is broad, but a frozen LLM simulator covers only one mode; the RL policy locks onto that mode and gives the narrow reply on real users it cannot serve. (b) Verbalized Sampling (inference-time): a single prompt asks the still-frozen simulator for several plausible user replies with likelihoods, so the policy sees varied reactions (acceptance, pushback, refusal) within a single rollout and cannot collapse onto a shortcut. (c) Co-Training (training-time): the simulator is no longer frozen and keeps drifting across training; the policy must keep generalising because the target mode it would memorise has already moved. Aligned LLMs favor typical responses under direct prompting (Jiang et al. 2025; Zhang et al. 2025a; Zhang et al. 2025b; GX-Chen et al. 2025); recent user-simulation studies confirm the same pattern, with LLM simulators reading as overly cooperative and stylistically uniform (Naous et al. 2025; Zhou et al. 2026; Mehri et al. 2026). We sharpen this into a definition tied to the policy’s training rollouts: at the simulator turns the policy actually visits, the simulator’s response distribution is mode-collapsed. Collapse on the training rollouts has three consequences we trace step by step. The policy gradient ends up close to one against a deterministic mode-user simulator (§3.2). Group-relative updates then ladder policy entropy down onto the narrow strategy that wins against the mode. §3.3 measures these predictions; §3.4 shows how Co-Training breaks the chain by making the simulator a moving target. 3.1 Definition and Hypothesis Mode collapse. At a simulator turn, the user response is sampled from ϕψ(⋅∣st,atπ) _ψ(· s_t,a_t^π). We call the most likely response at that turn the simulator’s mode: aϕ⋆(s,aπ)∈arg⁡maxaϕϕψ(aϕ∣s,aπ),a_φ (s,a^π)\;∈\; _a^φ _ψ(a^φ s,a^π), (3) and write ϵϕ(s,aπ)=1−ϕψ(aϕ⋆(s,aπ)∣s,aπ) _φ(s,a^π)=1- _ψ(a_φ (s,a^π) s,a^π) for the probability that the simulator deviates from its mode. Several recent works document strong mode concentration in aligned LLMs, often called mode collapse (Jiang et al. 2025; Zhang et al. 2025a; Zhang et al. 2025b; GX-Chen et al. 2025): small ϵϕ _φ means the simulator keeps emitting its mode. Definition 3.1 (Simulator collapse on the training rollouts). For a policy πθ _θ and a threshold ϵ⋆∈[0,1]ε ∈[0,1], we say the simulator is ϵ⋆ε -collapsed on the training rollouts if (st,atπ)∼(πθ,ϕψ)[ϵϕ(st,atπ)]≤ϵ⋆E_(s_t,a_t^π) ( _θ, _ψ)\! [ _φ(s_t,a_t^π) ]\;≤\;ε at the simulator turns visited by rollouts from (πθ,ϕψ)( _θ, _ψ). This definition is deliberately tied to the training distribution: a simulator can produce many different trajectories across a dataset and still behave almost deterministically at the histories the current policy actually visits. Collapse means the simulator’s per-turn distribution is narrow; prompts that change which behavior is modal don’t broaden it. We state ϵϕ _φ over exact token sequences for notation, but every result below also applies after measurable coarsening Π:aϕ↦bϕ :a^φ b^φ onto a behavior-class space (dialogue acts, strategy clusters), since TV distance to the modal point mass is non-increasing under projection (Remark B.1). The threshold ϵ⋆ε is the parameter the rest of the chain sharpens: Theorem 3.2 gives a gradient-bias bound that is tight when ϵ¯H(θ)≤ϵ⋆H ε_H(θ)≤ε H. The empirical proxy in Figure 19 (zero-variance batch fraction) is a one-sided diagnostic of training-signal degeneration, not a direct estimator of ϵϕ _φ: small ϵϕ _φ implies small reward variance via Lemma 3.3, but the converse can fail because sparse binary rewards or all-failure batches give zero variance too. A cleaner diagnostic would separate simulator-side from agent-side variance via Eq. 6; only the simulator-side term tracks ϵϕ _φ. Hypothesis. A mode-collapsed simulator breaks multi-agent RL into a fixed-user setting: the RL policy learns the strategy that wins against the simulator’s mode, and that strategy fails when other models or real users deviate from it. 3.2 Theory: How Simulator Collapse Impacts the Policy Mode collapse biases the policy gradient toward a deterministic mode-user objective (Theorem 3.2). It also kills simulator-side reward variance, so group-relative advantages rank samples by mode-exploit ability rather than user-robustness (Lemma 3.3). The policy gradient learns this signal, and policy mass concentrates geometrically onto the mode-exploit set AxA_x (Proposition 3.4, Corollary 3.5). The resulting low-entropy policy underperforms on real users with behaviors outside AxA_x (Proposition 3.6). Proofs are deferred to Appendix B. Collapse turns the gradient into a mode-user gradient. Let MϕM_φ be the dialogue POMDP induced by ϕφ, and MmodeM_mode the modified POMDP in which every simulator turn deterministically emits the mode aϕ⋆(st,atπ)a_φ (s_t,a_t^π). MmodeM_mode is determined by ϕψ _ψ alone, not by πθ _θ. Let Jmode(θ)J_mode(θ) be the corresponding objective. Theorem 3.2 (Simulator collapse induces mode-user optimization). Assume rewards are bounded in [0,Rmax][0,R_ ] and the trajectory-level policy score satisfies ∥∑t∇θlogπθ(atπ∣st)∥≤B _t _θ _θ(a_t^π s_t) ≤ B (e.g., under finite-length truncation and gradient clipping; see Appendix B.2). Couple πθ _θ in MϕM_φ and MmodeM_mode turn by turn (same task, same agent randomness, maximal coupling at each simulator turn). Define the accumulated collapse error along the rollout as ϵ¯H(θ)=⁡[∑t=1Hϵϕ(st,atπ)]. ε_H(θ)\;=\;E\! [\, _t=1^H _φ(s_t,a_t^π) ]. (4) Write PϕθP_φ^θ and PmodeθP_mode^θ for the joint trajectory distributions under πθ _θ in MϕM_φ and MmodeM_mode respectively. Then DTV(Pϕθ,Pmodeθ)≤ϵ¯H(θ)D_TV\! (P_φ^θ,\,P_mode^θ )≤ ε_H(θ), and ∥∇θJϕ(θ)−∇θJmode(θ)∥≤ 2BRmaxϵ¯H(θ). _θJ_φ(θ)- _θJ_mode(θ) \;≤\;2BR_ \, ε_H(θ). (5) Theorem 3.2 shows that the gradient does not vanish; it is biased, up to ϵ¯H(θ) ε_H(θ), toward the objective in which every user emits the mode aϕ⋆a_φ . The bound applies to the idealized REINFORCE gradient under a bounded trajectory-score assumption; the implemented update uses GRPO-style clipping, group-relative normalization, and gradient clipping (Appendix C.1), so the theorem is an analytic guide to the bias direction rather than a tight bound on the actual surrogate loss. The bound is informative when ϵ¯H≪1 ε_H 1; Figure 19’s zero-variance batch fraction climbing past 85%85\% is consistent with this regime at H=30,50H=30,50, with the proxy caveats above (Appendix B.4). What group-relative advantages measure. The same point shows up in the within-task reward variance that group-normalized RL z-scores. Let ξπ _π be agent-side randomness, ξU _U simulator-side randomness, and Rx=R⁡(x,ξπ,ξU)R_x=R(x, _π, _U). The law of total variance gives Var⁡[Rx∣x]=ξπ[VarξU(Rx∣x,ξπ)]⏟simulator-side contrast+Varξπ[ξU(Rx∣x,ξπ)]⏟agent-side contrast.Var[R_x x]= E_ _π\! [Var_ _U(R_x x, _π) ]_simulator-side contrast+ Var_ _π\! [E_ _U(R_x x, _π) ]_agent-side contrast. (6) Lemma 3.3 (Collapse removes simulator-side reward contrast). If the simulator’s trajectory is ϵH(x,ξπ) _H(x, _π)-close in TV to the mode trajectory at (x,ξπ)(x, _π), then VarξU(Rx∣x,ξπ)≤Rmax2ϵH(x,ξπ)Var_ _U(R_x x, _π)≤ R_ ^2\, _H(x, _π). When simulator-side variance vanishes, the z-scored advantage measures only agent-side variation. Group-relative RL then ranks samples by how well they exploit the simulator’s mode; user-robustness drops out of the comparison. Policy entropy collapses under a persistent mode advantage. Let Y be an agent-strategy abstraction at the level above tokens, with distribution qk(y∣x)q_k(y x) at update k. Intuitively, Y coarsens semantically-equivalent agent responses into a small set of strategies (e.g., “open with empathy”, “cite charity statistics”). This is the agent-side analog of the user-side coarsening in Definition 3.1. A strategy y exploits the simulator’s mode aϕ⋆a_φ if its mode-user value Qmode(x,y)Q_mode(x,y) is high, i.e., y accrues reward in the counterfactual MDP where the simulator deterministically emits aϕ⋆a_φ at every turn. Let AxA_x denote the set of these mode-exploit strategies, and let Δx>0 _x>0 be the mode-exploit gap: the smallest QmodeQ_mode advantage of any y∈Axy∈ A_x over any y′∉Axy ∉ A_x (Appendix B.6). By Theorem 3.2 the realized rollouts are dominated by trajectories in which the simulator emits aϕ⋆a_φ , so Δx _x quantifies the gap between the best mode-exploit strategy and the best non-exploit strategy on exactly the trajectories the agent sees during training. We analyze an idealized KL-regularized softmax update on qkq_k as a stylized model of the token-level GRPO step we run in practice; the proposition is a sufficient mechanism for the observed entropy collapse, not a direct theorem about the implemented optimizer. Proposition 3.4 (Mode advantage concentrates policy mass). Under a KL-regularized softmax policy-gradient step on qk(y∣x)q_k(y x) with learning rate η and bounded Q-estimation errors, the log-odds of AxA_x satisfy log⁡qk+1(Ax∣x)qk+1(Axc∣x)≥log⁡qk(Ax∣x)qk(Axc∣x)+gx, \! q_k+1(A_x x)q_k+1(A_x^c x)\;≥\; \! q_k(A_x x)q_k(A_x^c x)+g_x, (7) where gx>0g_x>0 depends on η, the gap Δx _x, and the simulator-collapse and estimation errors (Appendix B.6 gives exact constants). Corollary 3.5 (Entropy concentration). Iterating Eq. 7 gives qk(Ax∣x)≥11+1−q0(Ax∣x)q0(Ax∣x)e−kgx,q_k(A_x x)\;≥\; 11+ 1-q_0(A_x x)q_0(A_x x)\,e^-kg_x, (8) so the strategy distribution concentrates onto AxA_x geometrically fast in k. Token-level entropy, which we measure in §3.3, is the empirical counterpart of this strategy-level concentration only to the extent that the exploit strategies in AxA_x are themselves low-entropy text (fixed scripts, formulaic appeals); in that regime, strategy concentration shows up as a token-entropy collapse. We support this regime qualitatively by inspecting late-training within-batch rollouts, where the three transcripts from a single context become nearly word-for-word the same (Appendix F.9). From entropy collapse to transfer failure. By Corollary 3.5, the trained policy concentrates on AxA_x. This fails on real users whose behaviors are not handled by the exploit strategies in AxA_x. Proposition 3.6 (Deployment regret from missing user behaviors). Let BxB_x be a set of real-user behaviors requiring a strategy outside AxA_x, with real-user mass q⋆(x)=P⋆(Bx∣x)q_ (x)=P_ (B_x x). Suppose every ym∈Axy_m∈ A_x is worse than an adaptive strategy yby_b by at least Δxreal _x^real on BxB_x and by at most νx _x outside BxB_x, and the trained policy places probability at least 1−αx1- _x on AxA_x. Then J⋆(yb∣x)−J⋆(π^∣x)≥(1−αx)[Δxrealq⋆(x)−νx(1−q⋆(x))]−αxRmax.J_ (y_b x)-J_ ( π x)\;≥\;(1- _x) [ _x^real\,q_ (x)- _x(1-q_ (x)) ]- _xR_ . (9) The bound is positive when Δxrealq⋆(x)>νx(1−q⋆(x)) _x^real\,q_ (x)> _x(1-q_ (x)), i.e., missing behaviors are common enough that the adaptive strategy’s gain exceeds its off-behavior penalty. §3.3 tests this regime empirically: the collapsed policy fails when real users push back or apply constraints the training simulator rarely produced. 3.3 Empirical Evidence We examine these results empirically on Persuasion for Good (Wang et al. 2019) and τ2τ^2-bench (Barres et al. 2025). For each single-simulator run we track training reward, policy entropy, and eval reward on a held-out panel Φeval _eval of six simulators spanning seen and unseen families. Setup details are in Appendix C.4. Single-simulator RL exhibits simulator collapse in practice. We train against three frozen simulators of varying modal concentration (GPT-5-mini, Haiku-4.5, Gemini-3-Flash). Figure 3 shows the training results. Training reward climbs in every run, fastest for the most modal simulator. OOD eval peaks early and turns over, but the magnitudes differ sharply: the most modal simulator (Gemini-3-Flash) crashes below the untrained baseline, while the least modal (GPT-5-mini, the one we use in our main experiments) sees a gentler but still clear decline. Policy entropy crashes toward zero across all three runs. The failure is on the policy side: the policy learns the narrow exploit and fails to transfer to unseen simulators. The token-entropy pattern is consistent with the strategy-level concentration that Corollary 3.5 predicts, under the condition that the exploit strategies are themselves low-entropy (Remark B.4). Figure 3: Single-simulator RL exhibits simulator collapse. Three single-simulator REINFORCE runs, three seeds each, ±1σ± 1σ shading on OOD. Training reward climbs in every run (a), OOD eval peaks early and declines (b), and policy entropy collapses (c). The decoupling between (a) and (b) is simulator collapse passing through into policy collapse; (c) is the mechanism. Takeaway. The failure comes from the training environment, instead of the algorithm. Two solutions follow, at different points in the training loop. Verbalized Sampling (Zhang et al. 2025a) is the inference-time solution: we draw from a verbalized response distribution at each turn during rollout. Co-Training is the training-time solution: we update the simulator alongside the policy, so the mode the policy could lock onto shifts as training proceeds. The next section develops both solutions; their empirical comparison is in §4.2. 3.4 Breaking the Chain: Verbalized Sampling and Co-Training The collapse chain in §3.2 rests on two load-bearing assumptions. First, the per-turn collapse error ϵϕ(st,atπ) _φ(s_t,a_t^π) stays close to zero across the horizon, so ϵ¯H(θ) ε_H(θ) in Theorem 3.2 shrinks toward zero and ∇θJϕ _θJ_φ coincides with ∇θJmode _θJ_mode. Second, the modal-exploit set AxA_x stays fixed across training, so log-odds for AxA_x accumulate over k updates and Corollary 3.5 concentrates qkq_k on AxA_x at geometric rate gxg_x. The two fixes in this paper attack one assumption each. Under the reference-recovery assumption (DTV(pϕVS,P)≤ηD_TV(p^VS_φ,P)≤η; Proposition 3.7), Verbalized Sampling makes the policy gradient approximate the reference-user gradient instead of the mode-user gradient that Corollary 3.5 requires. Co-Training moves AxA_x at every step, so no policy iterate can stack log-odds toward a fixed target. Verbalized Sampling recovers the reference simulator distribution. A greedy query drives ϵϕ(s) _φ(s) toward zero. A verbalized query returns K candidate responses with verbalized probabilities; the resulting distribution pϕVS(⋅∣s)p^VS_φ(· s) approximates the simulator’s pre-RLHF reference distribution P(⋅∣s)P(· s) (Zhang et al. 2025a), since the distribution-level prompt verbalizes the response distribution that direct aligned prompting sharpens away. The closeness DTV(pϕVS,P)≤ηD_TV(p^VS_φ,P)≤η is an empirical assumption; Appendix F.5 discusses when it holds and when it fails. P comes from the simulator’s pretraining and is distinct from the real user population PrealP_real; the proposition below makes no claim about distance from real users, which the human study in Appendix E tests separately. Proposition 3.7 (Reference-gradient recovery under Verbalized Sampling). If DTV(pϕVS(⋅∣s,aπ),P(⋅∣s,aπ))≤η(s,aπ)D_TV(p^VS_φ(· s,a^π),\,P(· s,a^π))≤η(s,a^π) at every visited state along an H-turn rollout, then the trajectory distributions and policy gradients in MVSM_VS vs. the reference-user environment MrefM_ref satisfy DTV(PVSθ,Prefθ)≤η¯H(θ)D_TV(P^θ_VS,P^θ_ref)≤ η_H(θ) and ∥∇θJVS(θ)−∇θJref(θ)∥≤ 2BRmaxη¯H(θ), _θJ_VS(θ)- _θJ_ref(θ) \;≤\;2BR_ \, η_H(θ), where η¯H(θ)=⁡[∑t=1Hη⁡(st,atπ)] η_H(θ)=E [ _t=1^Hη(s_t,a_t^π) ]. This is the positive counterpart of Theorem 3.2: collapse induces a mode-user gradient; VS recovery induces a reference-user gradient. When P has non-trivial mass away from its mode and η is small, VS moves training out of the mode-oracle regime that Corollary 3.5 requires. Appendix F.5 proves the bound and discusses what the reference-recovery assumption does and does not buy; a separate γ-sharpening result there shows that direct aligned prompting exponentially suppresses tail behaviors while VS preserves them. Co-Training lets the simulator and policy co-evolve. We update the user simulator on its own turns of the same conversation, so both sides receive gradients from a single rollout. The simulator’s mode at each history shifts as training proceeds, and a strategy that exploited yesterday’s mode no longer wins against today’s: AxA_x is no longer constant, and the geometric concentration in Corollary 3.5 no longer applies (Appendix B.7 formalizes this via exclusive-lead counters). For the moving target to remain useful, the simulator must be trained with a reward that keeps it in the informative-variation regime (Remark B.10) rather than re-collapsing onto a different mode; task-specific reward choices are in §4.1 and an ablation showing both reward extremes underperform is in Appendix F.8. For binary rewards the curriculum targets success rate ≈0.5≈ 0.5, where within-batch variance p⁡(1−p)p(1-p) peaks at σ2=0.25σ^2=0.25 and group-relative advantages have the widest spread. 4 Experiments The theory makes three predictions we test in turn. (Q1) Single-simulator RL should collapse: OOD eval peaks early then declines as the policy locks onto a mode-exploit strategy that fails to transfer. (Q2) The two solutions should each recover the gradient signal at the layer they target: Verbalized Sampling by widening the simulator’s per-turn distribution, Co-Training by moving the mode the policy would chase. (Q3) Sampling the simulator from a pool of recent checkpoints should buy a further gain on top of Co-Training, and the size of that gain should depend on how much the pool preserves real variation. We also ask (Q4): do the LLM-panel gains transfer to real users? Section 4.2 answers Q1, Q2, and Q4 (the last on τ2τ^2-bench and P4G via a pre-registered human study); Appendix F.1 answers Q3 by isolating pool size and simulator reward. 4.1 Setup Tasks. Three multi-turn benchmarks: Persuasion for Good (P4G) (Wang et al. 2019), a persuader arguing for a charitable donation against a resistant donor (reward r=min⁡(donation/2,1)r= (donation/2,1)); τ2τ^2-bench (Barres et al. 2025), customer-service dialogues on retail or airline tasks with binary per-split success; and CooperBench (Khatua et al. 2026), two coding agents coordinating on a multi-step task with binary success. CooperBench uses Qwen3.5-9B and Qwen3.5-27B since smaller models cannot complete the tasks. P4G uses its donation-based adversarial reward and CooperBench its symmetric task-success reward; on τ2τ^2-bench the simulator is trained with a SPICE-style curriculum reward (Liu et al. 2025d) that targets within-batch variance σ2≈0.25σ^2≈ 0.25; this choice is essential, since adversarial and cooperative simulator rewards both collapse the simulator onto a new mode and drop eval reward (Appendix F.8). We evaluate every 16 steps and pick the best-mean-panel-score checkpoint. P4G and τ2τ^2-bench use a 6-simulator panel (3 seen during training plus 3 unseen families); CooperBench uses symmetric cross-play against Claude Haiku 4.5, GPT-5, and Gemini-3-Flash. Selecting on the full mean leaks partial training-simulator signal into selection; per-simulator and seen/unseen breakdowns are in Appendix C.4. Training. All methods share the policy update from §2. For Co-Training, both sides update from the same rollout, the simulator on a task-appropriate objective. The population variant additionally samples the active simulator from a pool of recent checkpoints (full buffer specification in Appendix C.1). The framework SCOPE implements all paradigms. Methods differ in per-step training compute: frozen-simulator methods (VS, Ensemble, Persona-Guided) update only the agent, while Co-Training and Population Co-Training update both sides on the same rollout. All comparisons in Tables 1 and 2 are at matched optimizer-step count rather than matched compute; per-step multipliers and total GPU-hours are in Appendix C.4. Table 1: User-simulator setting (P4G, τ2τ^2-bench). Best held-out-panel score across training; subscript = panel-std over six held-out simulators. P4G reward r=min⁡(donation/2,1)r= (donation/2,1); τ2τ^2-bench reports per-split success rate (%). Bold = best per model size; underline = runner-up. “N/A” on the P4G column for Persona-Guided: the P4G task already conditions on personas, so the baseline does not apply. The frozen ensemble uses K=3K=3 to match the main-text cost comparison; Population Co-Training uses K=5K=5 by default, and Appendix F.1 sweeps K∈1,3,5,10K∈\1,3,5,10\ showing the population benefit persists at each K. Per-method compute and memory breakdown is in Appendix C.4. P4G τ2τ^2-bench Model Method Reward ↑ Retail ↑ Airline ↑ Qwen3-4B-Instruct Base 0.216±0.03 40.4±3.1 24.0±2.8 RL (Single) 0.275±0.12 46.1±5.2 29.8±4.3 + Persona-Guided N/A 49.2±3.2 31.6±3.0 + Ensemble Models (K=3K=3) 0.394±0.04 57.1±3.4 40.1±3.6 + Verbalized Sampling 0.484±0.05 55.5±3.8 36.9±3.5 + Co-Training 0.438±0.05 60.5±3.9 44.4±4.0 + Population Co-Training 0.508±0.04 62.2±3.6 45.7±3.7 Qwen3-8B Base 0.253±0.03 48.1±3.0 30.2±2.9 RL (Single) 0.342±0.11 52.5±6.1 35.2±5.9 + Persona-Guided N/A 55.7±3.1 37.4±3.0 + Ensemble Models (K=3K=3) 0.450±0.04 62.4±3.3 43.6±3.5 + Verbalized Sampling 0.587±0.05 60.7±3.5 40.2±3.4 + Co-Training 0.556±0.05 66.1±3.7 48.2±3.8 + Population Co-Training 0.568±0.05 67.9±3.4 49.7±3.6 Baselines. Six paradigms under matched setup: Base (untrained); RL (Single) against GPT-5-mini, the standard recipe; Verbalized Sampling (Zhang et al. 2025a), sampling from the simulator’s verbalized response distribution; Ensemble Models, cycling K=3K=3 frozen models from different families; Co-Training, pairing the policy with a separately trained simulator; and Population Co-Training (ours), sampling the active simulator from a pool of recent checkpoints rather than the latest one alone. Plus a Persona-Guided simulator (Abdulhai et al. 2026), which conditions a single GPT-5-mini simulator on per-rollout personas. P4G’s task already conditions on personas, so Persona-Guided doesn’t apply here. 4.2 Simulator Collapse Reproduces; Both Fixes Recover It Figure 4: Both inference- and training-time solutions revive policy entropy. (a) P4G Eval Reward over training; (b) τ2τ^2-bench Retail Eval Success Rate; (c) τ2τ^2-bench Retail Policy Entropy. RL (Single) rises briefly and collapses below the untrained baseline on both eval panels while its entropy crashes to near zero. Verbalized Sampling, Ensemble (K=3K=3), Co-Training, and Population Co-Training improve eval and preserve entropy; the two Co-Training variants additionally exhibit the simulator-update kick pattern in the entropy panel. Persona-Guided (Abdulhai et al. 2026; τ2τ^2-bench only) sits between RL (Single) and the population methods. Curves average three seeds; eval shading is ±1σ± 1σ over six held-out evaluator models. Q1: Does single-simulator RL collapse? Yes, but the collapse is in the curve shape, not the headline number. Table 1 reports the best held-out checkpoint, and RL (Single)’s best is meaningfully above Base on every cell (at Qwen3-4B-Instruct: 46.146.1 vs 40.440.4 on τ2τ^2-Retail; 29.829.8 vs 24.024.0 on Airline; 0.2750.275 vs 0.2160.216 on P4G). The training curves in Figure 4 explain why: the best checkpoint is a transient peak that collapses back toward Base on τ2τ^2-Retail before training ends (Appendix Figure 21 shows the same collapse pattern). At end of training, RL (Single) and Persona-Guided sit within a few points of Base on both τ2τ^2 splits; the Table 1 best-checkpoint values reflect a transient peak, not steady-state behavior. The population methods stay at or above their peak throughout, so their best-checkpoint number is also their steady-state behavior. P4G is milder at both model sizes because the continuous donation reward preserves within-batch variance; the curve still degrades from its peak (Appendix Figure 20). The held-out panel is LLM-based; the real-user question is addressed by the human study in Appendix E. Q2: Do the two proposed solutions recover the signal, and at the layers the theory says? Both do, and the recovery is most informative when read side-by-side. On Qwen3-4B-Instruct, Verbalized Sampling lifts τ2τ^2-Retail from 46.146.1 to 55.555.5, Airline from 29.829.8 to 36.936.9, and P4G reward from 0.2750.275 to 0.4840.484. This is most of the available gain, recovered without retraining either side, and is consistent with the mechanism the theory assigns to VS: by querying a verbalized distribution at every simulator turn, VS keeps ϵϕ _φ above zero per turn so the modal-gradient term in Theorem 3.2 no longer dominates. Persona-Guided, a prompt-level baseline that conditions the simulator on per-task personas, gives a smaller lift (→49.246.1\!→\!49.2 Retail, →31.629.8\!→\!31.6 Airline): prompt-level diversity is a partial fix that does not close the gap to the simulator-level interventions. Co-Training acts on the complementary layer: it lifts Retail and Airline further to 60.560.5 and 44.444.4 by moving the mode the policy would lock onto across training steps, so Corollary 3.5’s geometric concentration never has a fixed target. Population Co-Training tops both at 62.262.2 and 45.745.7 and is the best method on every τ2τ^2 split at both model sizes and on P4G at 4B. The one exception is Qwen3-8B P4G, where VS (0.5870.587) edges past both Co-Training (0.5560.556) and Population Co-Training (0.5680.568): the continuous donation reward fits response-level Verbalized Sampling well enough that the within-simulator fix already captures the available signal. Table 2: Symmetric cooperation (CooperBench). Held-out success rate (%) against a 3-model partner panel (Haiku 4.5, GPT-5, Gemini-3-Flash); subscript = panel-std. Bold = best per model size; underline = runner-up. ∗Tinker with LoRA adapters (Qwen3.5-27B only); per-step compute differs from the other rows. CooperBench is symmetric: the asymmetric paradigm labels of Table 1 specialise here to cross-play (frozen partner), Self-play (single model in both roles), and Population self-play (rotating among recent self-play checkpoints). Method Qwen3.5-9B Qwen3.5-27B∗ Success ↑ Success ↑ Base 23.7±5.2 47.8±4.8 Cross-play (Haiku) 28.8±5.5 54.3±5.2 + Cross-play Ensemble (K=3K=3) 29.8±5.3 56.1±5.0 + Self-play 32.8±6.0 61.7±5.5 + Population self-play (ours) 33.6±5.7 62.4±5.3 Q3: Symmetric cooperation: cross-play hits a ceiling; co-evolution breaks through. Beyond the user-simulator setting (P4G, τ2τ^2-bench), CooperBench tests whether the same fixed-partner vs. co-evolving-partner mechanism extends to symmetric cooperation, with both sides drawn from the same role. Table 2 reports held-out success against a 3-model partner panel (Claude Haiku 4.5, GPT-5, Gemini-3-Flash). Frozen-partner cross-play plateaus at a ceiling set by the partner’s capacity; only self-play and population self-play break through, with the pool variant reaching the same peak faster. Detailed training dynamics for Qwen3.5-9B and Qwen3.5-27B are in Appendix Figures 23 and 24. Q4: Do the gains transfer to real users? We run a human study on τ2τ^2-bench (retail split) and Persuasion for Good with four conditions per task: Base, RL (Single), +Verbalized Sampling, and +Co-Training, collecting N=40N=40 Prolific participants per cell. Each session measures the task outcome and a survey is distributed on rating dialogue quality. Co-Training improves τ2τ^2 task outcome over RL (Single), and both Verbalized Sampling and Co-Training improve P4G dialogue naturalness. Full design, sample sizes, and analysis plan are in Appendix E. Table 3: Q4: Human study on τ2τ^2-bench and Persuasion for Good. We report τ2τ^2-bench task outcome from the objective evaluator ([0,1][0,1]), overall satisfaction on a 1–7 Likert scale, P4G donation amount, and P4G overall satisfaction. Bold = best per column; underline = second best per column. ∗p<0.05^*\,p<0.05, p∗⁣∗<0.01^**\,p<0.01 vs RL (Single). τ2τ^2-bench P4G Method Task ↑ Natural ↑ Donation ($) ↑ Natural ↑ Base 0.41±0.22 4.77±1.89 0.51±0.60 3.93±1.91 RL (Single) 0.43±0.31 5.11±1.63 0.46±0.72 3.21±1.76 + Verbalized Sampling 0.63±0.28∗⁣∗_± 0.28^** 5.38±1.57 0.74±0.59 4.33±1.78∗⁣∗_± 1.78^** + Co-Training 0.70±0.43∗⁣∗_± 0.43^** 5.50±1.81 0.69±0.64 4.45±1.67∗⁣∗_± 1.67^** 5 Related Work Mode collapse in LLMs. RLHF narrows LLM output distributions both empirically (Jiang et al. 2025; Zhang et al. 2025b; Zhang et al. 2025a; Yang et al. 2025b) and structurally: GX-Chen et al. 2025 prove that KL-regularized RL specifies a unimodal optimum by construction. Specialized cases include reasoning collapse in agentic RL (Wang et al. 2026b) and persona collapse in role-play (Xiao et al. 2026). We add simulator collapse: when the mode-collapsed LLM is the training environment, it starves the policy gradient. LLM-based user simulation for RL. LLM simulators are the de facto training environment for dialogue and agentic RL (Anthis et al. 2025; Abdulhai et al. 2025; Qian et al. 2025; Zhao et al. 2025a; Sun et al. 2025; Gandhi et al. 2026; Abdulhai et al. 2026), and increasingly the evaluation side too (Barres et al. 2025; Zhou et al. 2025a). Their limits are now well-documented: stronger assistants make worse simulators (Naous et al. 2025), simulators diverge from real users in preference (Zhou et al. 2026) and behavior distribution (Mehri et al. 2026), and inherit homogeneous cooperative bias from their base models (Chopra et al. 2026; Suh et al. 2026). Existing mitigations act on the simulator side: behavioral taxonomies (Shim et al. 2025), theory-of-mind objectives (Zhou et al. 2025a), curiosity rewards (Wan et al. 2025), finer credit assignment (Yu et al. 2025b; Qian et al. 2025), evolved persona generators (Chopra et al. 2026), or inference-time fixes (Yang et al. 2026). All optimize against a static simulator distribution. We instead replace it with a co-evolving population and trace the failure to a policy-side mechanism (Theorem 3.2, Corollary 3.5). Multi-agent RL and co-training. Self-play has driven gains in games (Silver et al. 2016; Berner et al. 2019) and in LLM training across text games (Liu et al. 2025c), corpus-grounded interaction (Liu et al. 2025d), and reasoning (Zhao et al. 2025b); Liao et al. 2024 note its diversity ceiling, motivating dual-model co-training (Ma et al. 2024; Feng et al. 2026; Acikgoz et al. 2026). Multi-turn RL has reached collaborative reasoning (Zhou et al. 2025b; Hong et al. 2025) and social tasks (Yu et al. 2025b; Tomlin et al. 2025) on short horizons (1–3 rounds). We extend co-training to long-horizon dialogue with population-based partner sampling. 6 Discussion and Conclusion We identify simulator collapse as a structural failure for LLMs in multi-agent RL: LLM simulators are mode-collapsed, an RL policy trained against such a simulator inherits that narrowness, the policy’s own entropy collapses onto the strategy that wins against the simulator’s mode, and the resulting low-entropy policy fails to transfer to unseen simulators or real users. From this we make three contributions. First, we formalize simulator collapse and show it is a structural failure of the training environment. Second, we give two complementary solutions: Verbalized Sampling at inference and Co-Training at training. We release SCOPE, an open framework that unifies multi-model rotation, self-play, and dual-model Co-Training behind one interface. Third, across Persuasion for Good, τ2τ^2-bench, and CooperBench, single-simulator RL’s held-out success peaks early then drops back toward the untrained baseline by end of training; both solutions close most of the gap, and Population Co-Training takes the strongest held-out task success. All three settings are text-only, two-agent, English, and LLM-panel evaluated; whether the mechanism extends to N-agent populations, multimodal environments, or non-English settings remains open. Both solutions are simple because the bottleneck is in the environment rather than the algorithm. Limitations, broader impact, and future work are in Appendix A. References DeepSeek-AI [2025] DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. Liu et al. [2025a] Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective, 2025a. URL https://arxiv.org/abs/2503.20783. Yu et al. [2025a] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. Dapo: An open-source llm reinforcement learning system at scale, 2025a. URL https://arxiv.org/abs/2503.14476. Feng et al. [2025] Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in llms, 2025. URL https://arxiv.org/abs/2504.11536. Jin et al. [2025] Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning, 2025. URL https://arxiv.org/abs/2503.09516. Yang et al. [2025a] John Yang, Kilian Lieret, Carlos E. Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. Swe-smith: Scaling data for software engineering agents, 2025a. URL https://arxiv.org/abs/2504.21798. Wei et al. [2025] Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I. Wang. Swe-rl: Advancing llm reasoning via reinforcement learning on open software evolution, 2025. URL https://arxiv.org/abs/2502.18449. Qian et al. [2025] Cheng Qian, Zuxin Liu, Akshara Prabhakar, Jielin Qiu, Zhiwei Liu, Haolin Chen, Shirley Kokane, Heng Ji, Weiran Yao, Shelby Heinecke, Silvio Savarese, Caiming Xiong, and Huan Wang. UserRL: Training Interactive User-Centric Agent via Reinforcement Learning, September 2025. URL http://arxiv.org/abs/2509.19736. arXiv:2509.19736 [cs]. Zhou et al. [2025a] Xuhui Zhou, Valerie Chen, Zora Zhiruo Wang, Graham Neubig, Maarten Sap, and Xingyao Wang. TOM-SWE: User Mental Modeling For Software Engineering Agents, October 2025a. URL http://arxiv.org/abs/2510.21903. arXiv:2510.21903 [cs]. Wu et al. [2026] Shirley Wu, Evelyn Choi, Arpandeep Khatua, Zhanghan Wang, Joy He-Yueya, Tharindu Cyril Weerasooriya, Wei Wei, Diyi Yang, Jure Leskovec, and James Zou. HumanLM: Simulating Users with State Alignment Beats Response Imitation, February 2026. URL https://arxiv.org/abs/2603.03303. Barres et al. [2025] Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. τ2τ^2-Bench: Evaluating Conversational Agents in a Dual-Control Environment, June 2025. URL http://arxiv.org/abs/2506.07982. arXiv:2506.07982 [cs]. Khatua et al. [2026] Arpandeep Khatua, Hao Zhu, Peter Tran, Arya Prabhudesai, Frederic Sadrieh, Johann K. Lieberwirth, Xinkai Yu, Yicheng Fu, Michael J. Ryan, Jiaxin Pei, and Diyi Yang. CooperBench: Why Coding Agents Cannot be Your Teammates Yet, January 2026. URL http://arxiv.org/abs/2601.13295. arXiv:2601.13295 [cs]. Wang et al. [2026a] Zora Zhiruo Wang, John Yang, Kilian Lieret, Alexa Tartaglini, Valerie Chen, Yuxiang Wei, Zijian Wang, Lingming Zhang, Karthik Narasimhan, Ludwig Schmidt, Graham Neubig, Daniel Fried, and Diyi Yang. Position: Humans are missing from ai coding agent research. https://zorazrw.github.io/files/position-haicode.pdf, 2026a. Wang et al. [2019] Xuewei Wang, Weiyan Shi, Richard Kim, Yoojung Oh, Sijia Yang, Jingwen Zhang, and Zhou Yu. Persuasion for good: Towards a personalized persuasive dialogue system for social good. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5635–5649, Florence, Italy, 2019. Association for Computational Linguistics. URL https://aclanthology.org/P19-1566/. Abdulhai et al. [2025] Marwa Abdulhai, Ryan Cheng, Donovan Clay, Tim Althoff, Sergey Levine, and Natasha Jaques. Consistently Simulating Human Personas with Multi-Turn Reinforcement Learning, October 2025. URL http://arxiv.org/abs/2511.00222. arXiv:2511.00222 [cs]. Park et al. [2023] Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, UIST ’23, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400701320. doi: 10.1145/3586183.3606763. URL https://doi.org/10.1145/3586183.3606763. Park et al. [2024] Joon Sung Park, Carolyn Q. Zou, Jonne Kamphorst, Niles Egan, Aaron Shaw, Benjamin Mako Hill, Carrie Cai, Meredith Ringel Morris, Percy Liang, Robb Willer, and Michael S. Bernstein. Llm agents grounded in self-reports enable general-purpose simulation of individuals, 2024. URL https://arxiv.org/abs/2411.10109. Yu et al. [2025b] Haofei Yu, Zhengyang Qi, Yining Zhao, Kolby Nottingham, Keyang Xuan, Bodhisattwa Prasad Majumder, Hao Zhu, Paul Pu Liang, and Jiaxuan You. Sotopia-RL: Reward Design for Social Intelligence, October 2025b. URL http://arxiv.org/abs/2508.03905. arXiv:2508.03905 [cs]. Abdulhai et al. [2023] Marwa Abdulhai, Isadora White, Charlie Snell, Charles Sun, Joey Hong, Yuexiang Zhai, Kelvin Xu, and Sergey Levine. Lmrl gym: Benchmarks for multi-turn reinforcement learning with language models, 2023. URL https://arxiv.org/abs/2311.18232. Jiang et al. [2025] Liwei Jiang, Yuanjun Chai, Margaret Li, Mickel Liu, Raymond Fok, Nouha Dziri, Yulia Tsvetkov, Maarten Sap, Alon Albalak, and Yejin Choi. Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond), October 2025. URL http://arxiv.org/abs/2510.22954. arXiv:2510.22954 [cs]. GX-Chen et al. [2025] Anthony GX-Chen, Jatin Prakash, Jeff Guo, Rob Fergus, and Rajesh Ranganath. KL-regularized reinforcement learning is designed to mode collapse, 2025. URL https://arxiv.org/abs/2510.20817. Zhang et al. [2025a] Jiayi Zhang, Simon Yu, Derek Chong, Anthony Sicilia, Michael R. Tomz, Christopher D. Manning, and Weiyan Shi. Verbalized sampling: How to mitigate mode collapse and unlock llm diversity, 2025a. URL https://arxiv.org/abs/2510.01171. MacDiarmid et al. [2025] Monte MacDiarmid, Benjamin Wright, Jonathan Uesato, Joe Benton, Jon Kutasov, Sara Price, Naia Bouscal, Sam Bowman, Trenton Bricken, Alex Cloud, Carson Denison, Johannes Gasteiger, Ryan Greenblatt, Jan Leike, Jack Lindsey, Vlad Mikulik, Ethan Perez, Alex Rodrigues, Drake Thomas, Albert Webson, Daniel Ziegler, and Evan Hubinger. Natural emergent misalignment from reward hacking in production RL, 2025. URL https://arxiv.org/abs/2511.18397. Liu et al. [2025b] Mickel Liu, Liwei Jiang, Yancheng Liang, Simon Shaolei Du, Yejin Choi, Tim Althoff, and Natasha Jaques. Chasing moving targets with online self-play reinforcement learning for safer language models, 2025b. Liu et al. [2025c] Bo Liu, Leon Guertler, Simon Yu, Zichen Liu, Penghui Qi, Daniel Balcells, Mickel Liu, Cheston Tan, Weiyan Shi, Min Lin, Wee Sun Lee, and Natasha Jaques. SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learning, July 2025c. URL http://arxiv.org/abs/2506.24119. arXiv:2506.24119 [cs]. Williams [1992] Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8:229–256, 1992. doi: 10.1007/BF00992696. Zhang et al. [2025b] Yiming Zhang, Harshita Diddee, Susan Holm, Hanchen Liu, Xinyue Liu, Vinay Samuel, Barry Wang, and Daphne Ippolito. Noveltybench: Evaluating language models for humanlike diversity, 2025b. URL https://arxiv.org/abs/2504.05228. Naous et al. [2025] Tarek Naous, Philippe Laban, Wei Xu, and Jennifer Neville. Flipping the Dialogue: Training and Evaluating User Language Models, October 2025. URL http://arxiv.org/abs/2510.06552. arXiv:2510.06552 [cs] version: 1. Zhou et al. [2026] Xuhui Zhou, Weiwei Sun, Qianou Ma, Yiqing Xie, Jiarui Liu, Weihua Du, Sean Welleck, Yiming Yang, Graham Neubig, Sherry Tongshuang Wu, and Maarten Sap. Mind the sim2real gap in user simulation for agentic tasks, 2026. URL https://arxiv.org/abs/2603.11245. Mehri et al. [2026] Shuhaib Mehri, Philippe Laban, Sumuk Shashidhar, Marwa Abdulhai, Sergey Levine, Michel Galley, and Dilek Hakkani-Tür. Measuring and mitigating the distributional gap between real and simulated user behaviors, May 2026. URL https://arxiv.org/abs/2605.07847. Liu et al. [2025d] Bo Liu, Chuanyang Jin, Seungone Kim, Weizhe Yuan, Wenting Zhao, Ilia Kulikov, Xian Li, Sainbayar Sukhbaatar, Jack Lanchantin, and Jason Weston. SPICE: Self-Play In Corpus Environments Improves Reasoning, October 2025d. URL http://arxiv.org/abs/2510.24684. arXiv:2510.24684 [cs]. Abdulhai et al. [2026] Marwa Abdulhai, Ryan Cheng, Aryansh Shrivastava, Aviral Kumar, and Sergey Levine. Hierarchical agenda reasoning for strategic multi-turn dialogue agents. In Workshop on Scaling Post-training for LLMs, 2026. URL https://openreview.net/forum?id=p144zx4bO0. Yang et al. [2025b] Chenghao Yang, Sida Li, and Ari Holtzman. Llm probability concentration: How alignment shrinks the generative horizon, 2025b. URL https://arxiv.org/abs/2506.17871. Wang et al. [2026b] Zihan Wang, Chi Gui, Xing Jin, Qineng Wang, Licheng Liu, Kangrui Wang, Shiqi Chen, Linjie Li, Zhengyuan Yang, Pingyue Zhang, Yiping Lu, Jiajun Wu, Li Fei-Fei, Lijuan Wang, Yejin Choi, and Manling Li. Ragen-2: Reasoning collapse in agentic rl, 2026b. URL https://arxiv.org/abs/2604.06268. Xiao et al. [2026] Yunze Xiao, Vivienne J. Zhang, Chenghao Yang, Ningshan Ma, Weihao Xuan, and Jen tse Huang. The chameleon’s limit: Investigating persona collapse and homogenization in large language models, 2026. URL https://arxiv.org/abs/2604.24698. Anthis et al. [2025] Jacy Reese Anthis, Ryan Liu, Sean M. Richardson, Austin C. Kozlowski, Bernard Koch, James Evans, Erik Brynjolfsson, and Michael Bernstein. Llm social simulations are a promising research method, 2025. URL https://arxiv.org/abs/2504.02234. Zhao et al. [2025a] Weikang Zhao, Xili Wang, Chengdi Ma, Lingbin Kong, Zhaohua Yang, Mingxiang Tuo, Xiaowei Shi, Yitao Zhai, and Xunliang Cai. MUA-RL: Multi-turn User-interacting Agent Reinforcement Learning for agentic tool use, August 2025a. URL http://arxiv.org/abs/2508.18669. arXiv:2508.18669 [cs]. Sun et al. [2025] Weiwei Sun, Xuhui Zhou, Weihua Du, Xingyao Wang, Sean Welleck, Graham Neubig, Maarten Sap, and Yiming Yang. Training Proactive and Personalized LLM Agents, November 2025. URL http://arxiv.org/abs/2511.02208. arXiv:2511.02208 [cs]. Gandhi et al. [2026] Kanishk Gandhi, Agam Bhatia, and Noah D. Goodman. Learning to Simulate Human Dialogue, January 2026. URL http://arxiv.org/abs/2601.04436. arXiv:2601.04436 [cs]. Chopra et al. [2026] Harshita Chopra, Kshitish Ghate, Aylin Caliskan, Tadayoshi Kohno, Chirag Shah, and Natasha Jaques. Beyond cooperative simulators: Generating realistic user personas for robust evaluation of llm agents, 2026. URL https://arxiv.org/abs/2605.12894. Suh et al. [2026] Joseph Suh, Ayush Raj, Minwoo Kang, and Serina Chang. Quantifying the utility of user simulators for building collaborative llm assistants, 2026. URL https://arxiv.org/abs/2605.09808. Shim et al. [2025] Jeonghoon Shim, Woojung Song, Cheyon Jin, Seungwon KooK, and Yohan Jo. Non-collaborative user simulators for tool agents, September 2025. URL https://arxiv.org/abs/2509.23124. Wan et al. [2025] Yanming Wan, Jiaxing Wu, Marwa Abdulhai, Lior Shani, and Natasha Jaques. Enhancing personalized multi-turn dialogue with curiosity reward, 2025. URL https://arxiv.org/abs/2504.03206. Yang et al. [2026] Shu Yang, Shenzhe Zhu, Hao Zhu, José Ramón Enríquez, Di Wang, Alex Pentland, Michiel A. Bakker, and Jiaxin Pei. Multi-user large language model agents, 2026. URL https://arxiv.org/abs/2604.08567. Silver et al. [2016] David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):484–489, 2016. doi: 10.1038/nature16961. Berner et al. [2019] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique P. d. O. Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680, 2019. Zhao et al. [2025b] Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Yang Yue, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, and Gao Huang. Absolute Zero: Reinforced Self-play Reasoning with Zero Data, October 2025b. URL http://arxiv.org/abs/2505.03335. arXiv:2505.03335 [cs]. Liao et al. [2024] Austen Liao, Nicholas Tomlin, and Dan Klein. Efficacy of language model self-play in non-zero-sum games, 2024. URL https://arxiv.org/abs/2406.18872. Ma et al. [2024] Hao Ma, Tianyi Hu, Zhiqiang Pu, Boyin Liu, Xiaolin Ai, Yanyan Liang, and Min Chen. Coevolving with the other you: Fine-tuning LLM with sequential cooperative multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, volume 37, pages 15497–15525, 2024. Feng et al. [2026] Lang Feng, Longtao Zheng, Shuo He, Fuxiang Zhang, and Bo An. Dr. MAS: Stable reinforcement learning for multi-agent LLM systems, 2026. Acikgoz et al. [2026] Emre Can Acikgoz, Cheng Qian, Jonas Hübotter, Heng Ji, Dilek Hakkani-Tür, and Gokhan Tur. Tool-R0: Self-evolving LLM agents for tool-learning from zero data, 2026. Zhou et al. [2025b] Yifei Zhou, Song Jiang, Yuandong Tian, Jason Weston, Sergey Levine, Sainbayar Sukhbaatar, and Xian Li. SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Tasks, March 2025b. URL http://arxiv.org/abs/2503.15478. arXiv:2503.15478 [cs]. Hong et al. [2025] Joey Hong, Kang Liu, Zhan Ling, Jiecao Chen, and Sergey Levine. Natural Language Actor-Critic: Scalable Off-Policy Learning in Language Space, December 2025. URL http://arxiv.org/abs/2512.04601. arXiv:2512.04601 [cs]. Tomlin et al. [2025] Nicholas Tomlin, Naitian Zhou, Eve Fleisig, Liangyuan Chen, Téa Wright, Lauren Vinh, Laura X. Ma, Seun Eisape, Ellie French, Tingting Du, Tianjiao Zhang, Alexander Koller, and Alane Suhr. Characterizing Language Use in a Collaborative Situated Game, December 2025. URL http://arxiv.org/abs/2512.03381. arXiv:2512.03381 [cs]. Rafailov et al. [2023] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, volume 36, 2023. Shao et al. [2024] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. Schulman et al. [2017] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. Brockman et al. [2016] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016. Zheng et al. [2024] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. Sglang: Efficient execution of structured language model programs. arXiv preprint arXiv:2312.07104, 2024. Shoeybi et al. [2019] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. CoRR, abs/1909.08053, 2019. URL http://arxiv.org/abs/1909.08053. Zhu et al. [2025] Zilin Zhu, Chengxing Xie, Xin Lv, and slime Contributors. slime: An llm post-training framework for rl scaling. https://github.com/THUDM/slime, 2025. GitHub repository. Corresponding author: Xin Lv. Sheng et al. [2024] Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. HybridFlow: A flexible and efficient RLHF framework. 2024. Zheng et al. [2026] Haizhong Zheng, Yizhuo Di, Jiahui Wang, Shuowei Jin, Xueshen Liu, Yongji Wu, Z. Morley Mao, Ion Stoica, Jiawei Zhao, and Beidi Chen. Astraflow: Dataflow-oriented reinforcement learning for agentic llms, 2026. URL https://arxiv.org/abs/2605.15565. Qwen Team [2026] Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. URL https://qwen.ai/blog?id=qwen3.5. Olmo et al. [2025] Team Olmo, :, Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, Jacob Morrison, Jake Poznanski, Kyle Lo, Luca Soldaini, Matt Jordan, Mayee Chen, Michael Noukhovitch, Nathan Lambert, Pete Walsh, Pradeep Dasigi, Robert Berry, Saumya Malik, Saurabh Shah, Scott Geng, Shane Arora, Shashank Gupta, Taira Anderson, Teng Xiao, Tyler Murray, Tyler Romero, Victoria Graf, Akari Asai, Akshita Bhagia, Alexander Wettig, Alisa Liu, Aman Rangapur, Chloe Anastasiades, Costa Huang, Dustin Schwenk, Harsh Trivedi, Ian Magnusson, Jaron Lochner, Jiacheng Liu, Lester James V. Miranda, Maarten Sap, Malia Morgan, Michael Schmitz, Michal Guerquin, Michael Wilson, Regan Huff, Ronan Le Bras, Rui Xin, Rulin Shao, Sam Skjonsberg, Shannon Zejiang Shen, Shuyue Stella Li, Tucker Wilde, Valentina Pyatkin, Will Merrill, Yapei Chang, Yuling Gu, Zhiyuan Zeng, Ashish Sabharwal, Luke Zettlemoyer, Pang Wei Koh, Ali Farhadi, Noah A. Smith, and Hannaneh Hajishirzi. Olmo 3, 2025. URL https://arxiv.org/abs/2512.13961. Appendix A Limitations, Broader Impact, and Future Work A.1 Limitations Fixed pool. The frozen pool’s diversity is bounded by whatever models we draw from, and the set stays fixed during training. Adaptive pool curation is left for future work. LLM evaluation panel. Our held-out panel is itself a set of aligned LLMs, so it shares RLHF-induced biases with the training simulators [Zhou et al. 2026]. The pre-registered human study on τ2τ^2-bench and Persuasion for Good (Appendix E) is the direct test of real-user transfer. Task-specific simulator reward. Co-Training depends on a simulator reward whose curriculum preserves cross-checkpoint variation (Appendix F.8). We give one such reward that works on the benchmarks we tested; we have not mapped what else would work. Compute overhead. Both interventions add compute over single-simulator RL, especially Co-Training where two models update on the same rollouts. This is the cost of escaping a structural failure: a frozen, mode-collapsed simulator is unlikely to produce a policy that transfers to unseen partners, so any fix has to move the simulator’s distribution at some point in the loop. Plausible cost reductions: a smaller simulator pool, amortized Verbalized Sampling across turns, or warm-started Co-Training from a single-simulator checkpoint. A.2 Broader Impact We release an open framework for population-based multi-agent RL that unifies heterogeneous simulator rotation, self-play, and Co-Training behind one interface. As RL moves toward multi-agent settings, the bottleneck is shifting from perfecting individual simulators to curating diverse populations, which calls for infrastructure that brings population-based training closer in cost to single-simulator training. The diagnostic chain in §3.2 names a failure mode that agentic RL pipelines can now check for, and that informs which simulator and which reward to choose at the start of a training run. A.3 Future Work Several extensions follow from our results. (i) Adaptive simulator populations: the buffer can be replaced with a learned curator that decides which past checkpoints to keep based on training-time signal. (i) Learned simulator-reward shaping: Appendix F.1 shows the curriculum reward matters more than the pool itself, so meta-learning a simulator reward that maximizes cross-checkpoint disagreement follows directly. (i) Beyond two-agent settings: extending SCOPE to N≥3N≥3 multi-agent populations and mixed cooperative/adversarial task mixtures should test how far the simulator-collapse mechanism generalizes. (iv) Other RLHF regimes: we conjecture analogous environment-collapse phenomena in reasoning, code, and tool-use RL whenever the verifier or grader is itself a mode-collapsed LLM. The diagnostic chain of §3.2 should transfer with minimal change. Appendix B Theory: Simulator Collapse This appendix gives full proofs of the theorems in Section 3.2, following the main-text chain. Simulator collapse turns the policy gradient into a mode-user gradient (Appendix B.4); simulator-side reward variance vanishes under the same coupling (Appendix B.5); group-relative updates then concentrate policy mass on a modal-exploit set (Appendix B.6); the trained policy fails when held-out users emit response types the simulator rarely produced (Appendix B.8); the mixture-gradient bound for the population-co-training extension follows by linearity (Appendix B.9). Appendix B.3 restates the γ-sharpening result of Zhang et al. 2025a, which motivates Definition 3.1 but is not used in any proof below. B.1 Notation summary The most frequently used quantities in §3 and this appendix. Symbol Meaning aϕ⋆(s,aπ)a_φ (s,a^π) simulator’s mode at (s,aπ)(s,a^π) ϵϕ(s,aπ) _φ(s,a^π) per-turn collapse error: 1−ϕψ(aϕ⋆∣s,aπ)1- _ψ(a_φ s,a^π) ϵ⋆ε collapse threshold (Definition 3.1) ϵ¯H(θ) ε_H(θ) accumulated collapse error along an H-turn rollout b,ℓturn,Bb, _turn,B per-token / per-turn / trajectory score bounds; B≤bℓturnHB≤ b\, _turn\,H Y,qk(y∣x)Y,q_k(y x) strategy abstraction; strategy distribution at update k Ax,Δx,gxA_x, _x,g_x modal-exploit set, mode-exploit gap, geometric concentration rate pϕVS,Pp^VS_φ,P verbalized simulator distribution; pre-RLHF reference distribution η⁡(s,aπ),η¯H(θ)η(s,a^π), η_H(θ) per-turn and accumulated reference-recovery TV error (Proposition 3.7) ρ,m,λρ,m,λ reference-mass on B, modal mass, per-behavior bound on B (Proposition F.1) Φ,ϕ¯,mk , φ,m_k checkpoint buffer, mixture simulator, per-checkpoint peak mass ϵ¯HΦ ε_H accumulated collapse error under population mixing B.2 Preliminaries We work in the POMDP setup of Section 2: each trajectory τ=(s0,a0π,a0ϕ,…,sT)τ=(s_0,a_0^π,a_0^φ,…,s_T) has terminal reward R⁡(τ)∈[0,Rmax]R(τ)∈[0,R_ ]. Write Pϕθ(τ)P_φ^θ(τ) for the trajectory distribution under (πθ,ϕ)( _θ,φ) and Jϕ(θ)=τ∼Pϕθ[R⁡(τ)]J_φ(θ)=E_τ P_φ^θ[R(τ)]. Policy-gradient identity. REINFORCE [Williams 1992] writes ∇θJϕ(θ)=τ∼Pϕθ[R⁡(τ)Sθ(τ)],Sθ(τ)=∑ℓ∈ℐπ(τ)∇θlogπθ(aℓ∣hℓ), _θJ_φ(θ)\;=\;E_τ P_φ^θ\! [R(τ)\,S_θ(τ) ], S_θ(τ)\;=\; _ _π(τ) _θ _θ(a_ h_ ), (10) where ℐπ(τ)I_π(τ) indexes the agent-decision tokens. With trajectory length L and per-token score bound ‖∇θlogπθ(aℓ∣hℓ)‖≤b\| _θ _θ(a_ h_ )\|≤ b, the trajectory score satisfies ‖Sθ(τ)‖≤Lb=:B\|S_θ(τ)\|≤ Lb=:B. Since each turn produces a bounded number of agent-decision tokens ℓturn _turn, the trajectory length satisfies L≤ℓturnHL≤ _turnH, so the trajectory-score bound B=Lb≤bℓturnHB=Lb≤ b\, _turn\,H is linear in the horizon H. In our training runs the bound is enforced by gradient clipping at norm 1.01.0 (Table 6). Trajectory TV under maximal coupling. For any two trajectory distributions P,QP,Q on T there exists a coupling ν on ×T×T such that Pr(τ,τ′)∼ν[τ≠τ′]=DTV(P,Q) _(τ,τ ) ν[τ≠τ ]=D_TV(P,Q). We instantiate this turn by turn: at simulator turn t, given a matched prefix, the maximal coupling between ϕψ(⋅∣st,atπ) _ψ(· s_t,a_t^π) and δaϕ⋆(st,atπ) _a_φ (s_t,a_t^π) disagrees with probability ϵϕ(st,atπ) _φ(s_t,a_t^π) (Definition 3.1). Bounded-integrand TV lemma. For any bounded vector-valued f with ‖f‖∞≤M\|f\|_∞≤ M and any two probability measures P,QP,Q, ∥P[f]−Q[f]∥≤ 2MDTV(P,Q). _P[f]-E_Q[f] \;≤\;2M\,D_TV(P,Q). (11) This follows from the variational form of TV applied coordinate-wise. KL-regularized RL closed form. For the objective maxπ[R]−βKL(π∥πref) _πE_π[R]-β\,KL(π\,\|\, _ref), the optimum is πβ⋆(y)∝πref(y)exp⁡(R⁡(y)/β) _β (y) _ref(y) (R(y)/β) [Rafailov et al. 2023, GX-Chen et al. 2025]. We use this only in Appendix B.3, to motivate Definition 3.1. Remark B.1 (Behavioral collapse as action-space coarsening). Formally, ϵϕ _φ in Definition 3.1 is defined over the simulator’s token-action distribution ϕψ(⋅∣s,aπ) _ψ(· s,a^π), so the literal quantity is concentration around the modal token sequence aϕ⋆a_φ ; the behavioral reading corresponds to a coarsening of this action space, such as a projection onto dialogue-act labels or strategy clusters. For any measurable projection Π of the action space, the TV distance from Π∗ϕψ _* _ψ to the corresponding modal point mass is non-increasing, so the gradient-bias bound of Theorem 3.2 continues to apply on the coarsened distribution. We do not commit to a specific coarsening in this paper; the transcript inspections in Appendix F.9 illustrate the qualitative behavioral patterns we have in mind. B.3 Simulator Mode Collapse via γ-Sharpening (Motivation) Why are aligned LLMs mode-collapsed in the first place? Zhang et al. 2025a trace this to typicality bias in RLHF: aligned LLMs inherit a preference for high-likelihood responses from their reward models, and KL-regularized RL compounds that bias into an exponential concentration of probability mass. We restate their observation as a direct specialization of the closed-form optimum of KL-regularized RL. The proposition below motivates Definition 3.1 but is not used in any proof in this appendix. Proposition B.2 (γ-Sharpening, following Zhang et al. 2025a). Let simulator ϕφ be trained via KL-regularized RLHF with reference ϕref _ref, KL penalty β>0β>0, and reward model rϕ(s,y)=rtrue(s,y)+αlog⁡ϕref(y∣s)+ϵ⁡(s)r_φ(s,y)\;=\;r_true(s,y)+α _ref(y s)+ε(s) (12) for some typicality-bias weight α>0α>0, true-quality term rtruer_true, and prompt-specific offset ϵ⁡(s)ε(s). Then ϕ∗(y∣s)∝ϕref(y∣s)γ⋅exp⁡(rtrue(s,y)/β),γ:= 1+α/β> 1.φ^*(y s)\; \; _ref(y s)^γ· \! (r_true(s,y)/β ), γ\;:=\;1+α/β\;>\;1. (13) Proof. Substituting (12) into the closed-form KL-regularized optimum Gβ(y)∝ϕref(y∣s)exp⁡(rϕ(s,y)/β)G_β(y) _ref(y s) (r_φ(s,y)/β) (Appendix B.2), ϕ∗(y∣s)=eϵ⁡(s)/βZ⁡(s)ϕref(y∣s)1+α/βexp⁡(rtrue(s,y)/β).φ^*(y s)= e^ε(s)/βZ(s) _ref(y s)^1+α/β \! (r_true(s,y)/β ). (14) The prompt-specific factor eϵ⁡(s)/βe^ε(s)/β absorbs into the partition function, leaving (13). Empirical typicality-bias estimates put α≈0.5α≈ 0.5–0.650.65 [Zhang et al. 2025a]; combined with standard β∈[0.01,0.1]β∈[0.01,0.1] this places γ in the 66–6666 range, where the mode carries essentially all the mass of ϕ∗φ^*. ∎ Proposition B.2 is the specialization; the broader structural pressure toward unimodal solutions in KL-regularized RL is also documented by GX-Chen et al. 2025, who show this concentration arises even without a typicality bias. Either route leads to the same conclusion. The proofs below only invoke the measurable condition that ϵϕ(st,atπ) _φ(s_t,a_t^π) is small at the simulator turns visited during training (Definition 3.1). B.4 Proof of Theorem 3.2 Step 1: Per-turn coupling. Couple the trajectory τ in MϕM_φ and τ′τ in MmodeM_mode as follows. Both runs use the same task and the same agent randomness. At each simulator turn t, given a matched prefix, sample (atϕ,atϕ,⋆)(a_t^φ,a_t^φ, ) from the maximal coupling between ϕψ(⋅∣st,atπ) _ψ(· s_t,a_t^π) and δaϕ⋆(st,atπ) _a_φ (s_t,a_t^π); under maximal coupling the two samples agree with probability 1−ϵϕ(st,atπ)1- _φ(s_t,a_t^π). Once they disagree, the prefixes decouple and the remaining simulator turns are sampled independently. Let D=[τ≠τ′]D=I[τ≠τ ]. By the union bound over simulator turns, Pr[D=1]≤[∑t=1Hϵϕ(st,atπ)]=ϵ¯H(θ). [D=1]\;≤\;E\! [\, _t=1^H _φ(s_t,a_t^π) ]\;=\; ε_H(θ). (15) By the coupling characterization of TV, DTV(Pϕθ,Pmodeθ)≤Pr[D=1]≤ϵ¯H(θ),D_TV\! (P_φ^θ,\,P_mode^θ )\;≤\; [D=1]\;≤\; ε_H(θ), (16) which proves the trajectory-TV bound stated in Theorem 3.2. Step 2: From trajectory TV to gradient norm. Apply (10) to both objectives: ∇θJϕ(θ)−∇θJmode(θ)=τ∼Pϕθ[R⁡(τ)Sθ(τ)]−τ∼Pmodeθ[R⁡(τ)Sθ(τ)]. _θJ_φ(θ)- _θJ_mode(θ)\;=\;E_τ P_φ^θ\! [R(τ)\,S_θ(τ) ]-E_τ P_mode^θ\! [R(τ)\,S_θ(τ) ]. The integrand f⁡(τ)=R⁡(τ)Sθ(τ)f(τ)=R(τ)\,S_θ(τ) satisfies ‖f⁡(τ)‖≤Rmax‖Sθ(τ)‖≤RmaxB\|f(τ)\|≤ R_ \,\|S_θ(τ)\|≤ R_ B. Applying the bounded-integrand TV bound (11) with M=RmaxBM=R_ B together with the trajectory-TV bound from Step 1, ∥∇θJϕ(θ)−∇θJmode(θ)∥≤ 2RmaxB⋅DTV(Pϕθ,Pmodeθ)≤ 2BRmaxϵ¯H(θ), _θJ_φ(θ)- _θJ_mode(θ) \;≤\;2R_ B· D_TV\! (P_φ^θ,P_mode^θ )\;≤\;2BR_ \, ε_H(θ), which is (5). The specialization ϵ¯H(θ)≤Hϵ ε_H(θ)≤ Hε when ϵϕ(st,atπ)≤ϵ _φ(s_t,a_t^π)≤ε on all visited turns follows immediately. □ Regime where the bound is informative. Substituting B≤bℓturnHB≤ b\, _turn\,H makes the horizon dependence explicit: the gradient bound is at most 2bℓturnRmax⋅H⋅ϵ¯H(θ)2\,b\, _turn\,R_ · H· ε_H(θ). When the per-turn collapse error is roughly constant in t, ϵ¯H≈ϵavg⋅H ε_H≈ _avg· H, so the bound scales as O⁡(H2)O(H^2) in horizon at a fixed per-turn collapse rate. For τ2τ^2-bench at H=30H=30 the horizon prefactor multiplying ϵ¯H ε_H is 30⋅bℓturnRmax30· b\, _turn\,R_ , while for CooperBench at H=50H=50 it is 50⋅bℓturnRmax50· b\, _turn\,R_ , so at the same per-turn collapse rate the CooperBench slack is larger. The bound is informative when ϵ¯H≪1 ε_H 1, equivalently when the average per-turn deviation ϵ¯H/H ε_H/H falls below 1/H1/H. Figure 19 measures this regime empirically via the zero-variance batch fraction, which climbs from 60%60\% to over 85%85\% during single-simulator training. Outside the regime the bound loosens. Both Verbalized Sampling and Co-Training push the system out of strong collapse by keeping ϵϕ _φ above zero per turn. B.5 Proof of Lemma 3.3 Fix x and ξπ _π. Under the same coupling as Appendix B.4 (now with policy randomness fixed at ξπ _π), the simulator trajectory τ and the modal trajectory τ⋆τ agree except on an event of probability at most ϵH(x,ξπ) _H(x, _π). When they agree, R⁡(τ)=R⁡(τ⋆)=:cR(τ)=R(τ )=:c, which is constant given (x,ξπ)(x, _π). For any random variable X∈[0,Rmax]X∈[0,R_ ] that equals a constant c except on an event cA^c of probability at most p, Var⁡(X)≤⁡[(X−c)2]=⁡[(X−c)2[c]]≤Rmax2Pr⁡[c]≤Rmax2p.Var(X)\;≤\;E\! [(X-c)^2 ]\;=\;E\! [(X-c)^2\,I[A^c] ]\;≤\;R_ ^2\, [A^c]\;≤\;R_ ^2\,p. Setting X=RxX=R_x and p=ϵH(x,ξπ)p= _H(x, _π) yields the bound in Lemma 3.3. Taking expectation over ξπ _π gives the marginal version ξπ[VarξU(Rx∣x,ξπ)]≤Rmax2ξπ[ϵH]E_ _π[Var_ _U(R_x x, _π)]≤ R_ ^2\,E_ _π[ _H]. □ B.6 Proofs of Proposition 3.4 and Corollary 3.5 We restate the auxiliary assumption and the policy-gradient step that Proposition 3.4 requires, then give the proof. Assumption B.3 (Mode-exploit gap). For task x there is a set AxA_x and gap Δx>0 _x>0 such that for every y∈Axy∈ A_x and y′∉Axy ∉ A_x, Qmode(x,y)≥Qmode(x,y′)+ΔxQ_mode(x,y)≥ Q_mode(x,y )+ _x. The mode-user value Qmode(x,y)=[R(τ)∣x,Y=y,Mmode]Q_mode(x,y)=E[R(τ) x,Y=y,M_mode] is the expected return when the simulator always emits aϕ⋆a_φ . The assumption says the collapsed simulator rewards a narrow family of agent strategies more than the alternatives (in persuasion, a fixed donation script; in customer service, a shortcut that extracts information from an overly helpful user). We use the standard log-odds form of a KL-regularized softmax policy-gradient step: positive estimated advantage increases the relative log probability, up to bounded optimization error ρx _x: log⁡qk+1(y∣x)qk+1(y′∣x)≥log⁡qk(y∣x)qk(y′∣x)+η⁡(Q^ϕ(x,y)−Q^ϕ(x,y′))−ρx. q_k+1(y x)q_k+1(y x)\;≥\; q_k(y x)q_k(y x)+η ( Q_φ(x,y)- Q_φ(x,y ) )- _x. (17) Define gx=η⁡(Δx−2Rmaxϵx−2ζx)−ρxg_x=η( _x-2R_ _x-2 _x)- _x, where ϵx _x and ζx _x bound the simulator-collapse and Q-estimation errors at task x. Proof of Proposition 3.4. Fix y∈Axy∈ A_x and y′∉Axy ∉ A_x. Combining the assumed errors |Qϕ−Qmode|≤Rmaxϵx|Q_φ-Q_mode|≤ R_ _x and |Q^ϕ−Qϕ|≤ζx| Q_φ-Q_φ|≤ _x with the mode-exploit gap (Assumption B.3), Q^ϕ(x,y)−Q^ϕ(x,y′) Q_φ(x,y)- Q_φ(x,y )\; ≥Qϕ(x,y)−Qϕ(x,y′)−2ζx ≥\;Q_φ(x,y)-Q_φ(x,y )-2 _x ≥(Qmode(x,y)−Qmode(x,y′))−2Rmaxϵx−2ζx ≥\; (Q_mode(x,y)-Q_mode(x,y ) )-2R_ _x-2 _x ≥Δx−2Rmaxϵx−2ζx. ≥\; _x-2R_ _x-2 _x. Plugging into the softmax / KL-constrained log-odds step (17), log⁡qk+1(y∣x)qk+1(y′∣x)≥log⁡qk(y∣x)qk(y′∣x)+gx,gx:=η⁡(Δx−2Rmaxϵx−2ζx)−ρx. q_k+1(y x)q_k+1(y x)\;≥\; q_k(y x)q_k(y x)+g_x, g_x:=η( _x-2R_ _x-2 _x)- _x. (18) Eq. (18) holds for every pair (y,y′)(y,y ) with y∈Axy∈ A_x and y′∉Axy ∉ A_x, so the pairwise ratio satisfies qk+1(y)/qk+1(y′)≥egxqk(y)/qk(y′)q_k+1(y)/q_k+1(y )≥ e^g_x\,q_k(y)/q_k(y ). Summing the numerator over y∈Axy∈ A_x and the denominator over y′∉Axy ∉ A_x gives qk+1(Ax∣x)qk(Axc∣x)≥egxqk(Ax∣x)qk+1(Axc∣x),q_k+1(A_x x)\,q_k(A_x^c x)\;≥\;e^g_x\,q_k(A_x x)\,q_k+1(A_x^c x), which is the set-level inequality log⁡qk+1(Ax∣x)qk+1(Axc∣x)≥log⁡qk(Ax∣x)qk(Axc∣x)+gx, q_k+1(A_x x)q_k+1(A_x^c x)\;≥\; q_k(A_x x)q_k(A_x^c x)+g_x, matching (7). The update strictly increases qk(Ax∣x)q_k(A_x x) whenever gx>0g_x>0. □ Proof of Corollary 3.5. Let Lk=log⁡qk(Ax∣x)qk(Axc∣x)L_k= q_k(A_x x)q_k(A_x^c x) and gx=η⁡(Δx−2Rmaxϵx−2ζx)−ρx>0g_x=η( _x-2R_ _x-2 _x)- _x>0 by assumption. Proposition 3.4 gives Lk+1≥Lk+gxL_k+1≥ L_k+g_x, so by induction Lk≥L0+kgxL_k≥ L_0+kg_x. Converting log-odds back to mass via the logistic σ⁡(t)=1/(1+e−t)σ(t)=1/(1+e^-t), qk(Ax∣x)=σ⁡(Lk)≥σ⁡(L0+kgx)=11+1−q0(Ax∣x)q0(Ax∣x)e−kgx,q_k(A_x x)\;=\;σ(L_k)\;≥\;σ(L_0+kg_x)\;=\; 11+ 1-q_0(A_x x)q_0(A_x x)\,e^-kg_x, which is (8). The right-hand side approaches 11 geometrically in k with rate gxg_x. □ Remark B.4 (Strategy entropy vs. token entropy). Corollary 3.5 is a statement about strategy entropy, while Figure 19 plots token-level entropy. The two are linked by H⁡(S∣x)=H⁡(Y∣x)+H⁡(S∣Y,x)H(S x)=H(Y x)+H(S Y,x), where S is the token sequence and Y=fx(S)Y=f_x(S) is its strategy cluster. Token-level entropy collapse therefore lower-bounds strategy concentration only conditionally: strategy concentration implies a token-entropy drop when the residual term H⁡(S∣Y,x)H(S Y,x) is small, i.e., when the exploit strategies in AxA_x are themselves low-entropy text patterns. Appendix F.9 reports late-training within-batch rollouts that are nearly word-for-word the same, consistent with this regime; a quantitative within-batch overlap study is left for future work. B.7 Co-Training breaks geometric concentration The geometric concentration in Corollary 3.5 assumes the mode-exploit set AxA_x is fixed across training. Under Co-Training, Ax(k)A_x^(k) shifts with each simulator update. This subsection formalizes why the shifting breaks the concentration: the log-odds growth in Proposition 3.4 applies to the current exploit set Ax(k)A_x^(k), so the net log-odds for any single strategy y depends on how often y has an exclusive membership lead over alternatives. Exclusive-lead counters. For a pair of strategies (y,y′)(y,y ), define the exclusive-lead counters after K updates as NK+(y,y′)=|k<K:y∈Ax(k),y′∉Ax(k)|,NK−(y,y′)=|k<K:y∉Ax(k),y′∈Ax(k)|.N_K^+(y,y )=|\k<K:y∈ A_x^(k),\ y ∉ A_x^(k)\|, N_K^-(y,y )=|\k<K:y∉ A_x^(k),\ y ∈ A_x^(k)\|. NK+N_K^+ counts updates where y is in the exploit set and y′y is not, so y’s log-odds grow by gxg_x via Proposition 3.4. NK−N_K^- counts the reverse. Lemma B.5 (Net log-odds under shifting exploit set). Under the same softmax update as Proposition 3.4 with mode-exploit gap Δx _x, the pairwise log-odds after K updates satisfy log⁡qK(y∣x)qK(y′∣x)≥log⁡q0(y∣x)q0(y′∣x)+gx⋅(NK+(y,y′)−NK−(y,y′)). \! q_K(y x)q_K(y x)\;≥\; \! q_0(y x)q_0(y x)\;+\;g_x· (N_K^+(y,y )-N_K^-(y,y ) ). Proof. At each step k where y∈Ax(k)y∈ A_x^(k) and y′∉Ax(k)y ∉ A_x^(k), Proposition 3.4 gives log⁡[qk+1(y)/qk+1(y′)]≥log⁡[qk(y)/qk(y′)]+gx [q_k+1(y)/q_k+1(y )]≥ [q_k(y)/q_k(y )]+g_x. At each step where y∉Ax(k)y∉ A_x^(k) and y′∈Ax(k)y ∈ A_x^(k), the symmetric inequality (swap y,y′y,y in Proposition 3.4) gives a −gx-g_x contribution. Steps where both or neither are in Ax(k)A_x^(k) give no inequality. Summing across K updates yields the bound. □ ∎ Corollary B.6 (No exclusive lead, no concentration). If for every pair (y,y′)(y,y ) the expected exclusive lead satisfies ⁡[NK+(y,y′)−NK−(y,y′)]=o⁡(K)E[N_K^+(y,y )-N_K^-(y,y )]=o(K), then ⁡[log⁡(qK(y)/qK(y′))]−log⁡(q0(y)/q0(y′))=o⁡(K)⋅gxE\! [ (q_K(y)/q_K(y )) ]- (q_0(y)/q_0(y ))=o(K)· g_x, and the policy distribution qK\q_K\ cannot concentrate on any single strategy at geometric rate. When Co-Training satisfies the no-concentration condition. A natural sufficient model: at each step the simulator update independently re-randomizes the exploit set with the same marginal across strategies. Under this i.i.d. shift, ⁡[NK+]=⁡[NK−]E[N_K^+]=E[N_K^-] for every pair, so ⁡[NK+−NK−]=0E[N_K^+-N_K^-]=0 and Corollary B.6 applies. The informative-variation criterion (Remark B.10) is the empirical condition for the i.i.d. model to be a reasonable approximation: it prevents the simulator from re-collapsing on the same mode across steps. The data-bound diagnostic that would directly test this is the per-step exploit-set overlap |Ax(k)∩Ax(k+1)||A_x^(k)∩ A_x^(k+1)|; we defer measurement to future work. B.8 Proofs of Lemma B.7 and Proposition 3.6 We first restate Lemma B.7 and the missing-behavior notation it uses. Lemma B.7 (Finite-sample coverage of user behaviors). For task x, let BxB_x be a set of real-user behaviors that require a strategy outside AxA_x, and let qϕ(x)=Pϕ(Bx∣x)q_φ(x)=P_φ(B_x x) be the simulator’s mass on BxB_x. With G independent simulator rollouts on task x, the probability that the group contains at least one behavior from BxB_x is 1−(1−qϕ(x))G≤Gqϕ(x)1-(1-q_φ(x))^G≤ G\,q_φ(x). Proof of Lemma B.7. The G rollouts on task x produce independent simulator responses with Pr[Zi∈Bx]=qϕ(x) [Z_i∈ B_x]=q_φ(x). The probability that all G miss BxB_x is (1−qϕ(x))G(1-q_φ(x))^G, so Pr[∃i≤G:Zi∈Bx]= 1−(1−qϕ(x))G≤Gqϕ(x), [∃ i≤ G:Z_i∈ B_x]\;=\;1-(1-q_φ(x))^G\;≤\;G\,q_φ(x), where the upper bound is Bernoulli’s inequality (1−p)G≥1−Gp(1-p)^G≥ 1-Gp for p∈[0,1]p∈[0,1]. □ Proof of Proposition 3.6. Write J⋆(y∣x)=z∼P⋆(⋅∣x)[rx(y,z)]J_ (y x)=E_z P_ (· x)[r_x(y,z)] and let π π be the trained policy. By assumption PrY∼π^(⋅∣x)[Y∈Ax]≥1−αx _Y π(· x)[Y∈ A_x]≥ 1- _x. Step 1: Per-strategy regret on the exploit set. For any ym∈Axy_m∈ A_x, J⋆(yb∣x)−J⋆(ym∣x) J_ (y_b x)-J_ (y_m x) =q⋆(x)z∈Bx[rx(yb,z)−rx(ym,z)]+(1−q⋆(x))z∉Bx[rx(yb,z)−rx(ym,z)] =q_ (x)\,E_z∈ B_x\! [r_x(y_b,z)-r_x(y_m,z) ]+(1-q_ (x))\,E_z∉ B_x\! [r_x(y_b,z)-r_x(y_m,z) ] ≥q⋆(x)Δxreal−(1−q⋆(x))νx, ≥ q_ (x)\, _x^real-(1-q_ (x))\, _x, using rx(yb,z)−rx(ym,z)≥Δxrealr_x(y_b,z)-r_x(y_m,z)≥ _x^real on BxB_x and ≥−νx≥- _x off BxB_x. Step 2: Aggregate over the policy’s two components. Decompose J⋆(π^∣x)=Pr[π^∈Ax]Y∼π^[J⋆(Y∣x)∣Y∈Ax]+Pr[π^∈Axc]Y∼π^[J⋆(Y∣x)∣Y∈Axc].J_ ( π x)\;=\; [ π∈ A_x]\,E_Y π\! [J_ (Y x) Y∈ A_x ]\;+\; [ π∈ A_x^c]\,E_Y π\! [J_ (Y x) Y∈ A_x^c ]. Step 1 gives Y∼π^[J⋆(Y∣x)∣Y∈Ax]≤J⋆(yb∣x)−(Δxrealq⋆(x)−νx(1−q⋆(x)))E_Y π[J_ (Y x) Y∈ A_x]≤ J_ (y_b x)- ( _x^real\,q_ (x)- _x(1-q_ (x)) ); the off-exploit component is bounded above by RmaxR_ . Therefore J⋆(π^∣x) J_ ( π x) ≤(1−αx)[J⋆(yb∣x)−Δxrealq⋆(x)+νx(1−q⋆(x))]+αxRmax ≤(1- _x) [J_ (y_b x)- _x^real\,q_ (x)+ _x(1-q_ (x)) ]+ _xR_ ≤J⋆(yb∣x)−(1−αx)[Δxrealq⋆(x)−νx(1−q⋆(x))]+αxRmax, ≤ J_ (y_b x)-(1- _x) [ _x^real\,q_ (x)- _x(1-q_ (x)) ]+ _xR_ , using J⋆(yb∣x)≤RmaxJ_ (y_b x)≤ R_ . Rearranging, J⋆(yb∣x)−J⋆(π^∣x)≥(1−αx)[Δxrealq⋆(x)−νx(1−q⋆(x))]−αxRmax,J_ (y_b x)-J_ ( π x)\;≥\;(1- _x) [ _x^real\,q_ (x)- _x(1-q_ (x)) ]- _xR_ , which is (9). □ The bound is informative whenever the trained policy concentrates on AxA_x (αx _x small), real users put non-trivial mass on BxB_x (q⋆(x)q_ (x) not too small), and the adaptive strategy gap Δxreal _x^real exceeds the off-BxB_x penalty νx _x. These three conditions correspond to the qualitative-transcript pattern in Section 4.2: the trained policy uses one strategy, real users sometimes deviate, and an adaptive response to the deviation outperforms the strategy. B.9 Population Gradient and Coverage We restate the formal mixture-gradient bound, give its proof, then state the coverage interpretation and the informative-variation criterion for the simulator reward. All three were summarized in §3.4 and deferred here. Proposition B.8 (Mixture gradient averages over recent simulator modes). If each ϕk _k in the buffer is mode-collapsed around its own mode ak⋆a_k along the training rollouts with accumulated error ϵ¯H,k(θ) ε_H,k(θ), and Jmode,kJ_mode,k is the objective that always returns ak⋆a_k , then for the mixture JΦJ_ with weights wk=1/Kw_k=1/K, ∥∇θJΦ(θ)−∑kwk∇θJmode,k(θ)∥≤ 2BRmax∑kwkϵ¯H,k(θ). _θJ_ (θ)- _kw_k _θJ_mode,k(θ) \;≤\;2BR_ \, _kw_k\, ε_H,k(θ). (19) Proof of Proposition B.8. By linearity of the policy-gradient identity in the simulator distribution, ∇θJΦ(θ)=∑k=1Kwk∇θJϕk(θ). _θJ_ (θ)\;=\; _k=1^Kw_k\, _θJ_ _k(θ). Applying Theorem 3.2 to each ϕk _k, ∥∇θJϕk(θ)−∇θJmode,k(θ)∥≤ 2BRmaxϵ¯H,k(θ). _θJ_ _k(θ)- _θJ_mode,k(θ) \;≤\;2BR_ \, ε_H,k(θ). Combining via the triangle inequality on ∑kwk(∇Jϕk−∇Jmode,k) _kw_k (∇ J_ _k-∇ J_mode,k ), ∥∇θJΦ(θ)−∑kwk∇θJmode,k(θ)∥≤∑kwk∥∇θJϕk(θ)−∇θJmode,k(θ)∥≤ 2BRmax∑kwkϵ¯H,k(θ), _θJ_ (θ)- _kw_k\, _θJ_mode,k(θ) \;≤\; _kw_k\, _θJ_ _k(θ)- _θJ_mode,k(θ) \;≤\;2BR_ _kw_k\, ε_H,k(θ), which is (19). □ The bound is a corollary; the diversity claim is Lemma B.9. Proposition B.8 follows from Theorem 3.2 by linearity in the simulator distribution and the triangle inequality. It says only that the population gradient is biased toward the average of the modal-user gradients with average accumulated collapse error. It does not, on its own, say that this average target is structurally easier to fit than any single mode. The formal diversity claim, why the mixture’s per-turn collapse error is bounded below by a K-dependent floor, is in the lemma below. Lemma B.9 (Population mixing raises per-turn collapse error). Let the buffer Φ=ϕkk=1K =\ _k\_k=1^K induce conditional distributions ϕk(⋅∣s) _k(· s) at state s with peak masses mk(s):=maxa⁡ϕk(a∣s)m_k(s):= _a _k(a s), and let ϕ¯=∑kwkϕk φ= _kw_k\, _k be the mixture. The mixture’s peak mass satisfies maxa⁡ϕ¯(a∣s)≤∑k=1Kwkmk(s), _a φ(a s)\;≤\; _k=1^Kw_k\,m_k(s), (20) with equality only when all ϕk _k peak at a common action. In the fully-collapsed disjoint-mode case (ϕk _k deterministic with pairwise-distinct modes aϕ⋆,kk=1K\a_φ ,k\_k=1^K) under uniform weights wk=1/Kw_k=1/K, the bound is tight and the per-turn collapse error is ϵϕ¯(s):= 1−maxa⁡ϕ¯(a∣s)= 1−1K. _ φ(s)\;:=\;1- _a φ(a s)\;=\;1- 1K. (21) The 1−1/K1-1/K floor is the best-case bound under pairwise-disjoint modes; neighboring FIFO checkpoints overlap heavily in practice, so the realized error sits closer to the single-checkpoint floor ϵϕk(s)=1−mk(s) _ _k(s)=1-m_k(s). The gap population mixing exploits is the distance between consecutive checkpoints’ modes, which the curriculum-rewarded simulator update keeps positive. Proof of Lemma B.9. For any fixed a, ϕ¯(a∣s)=∑kwkϕk(a∣s)≤∑kwkmk(s) φ(a s)= _kw_k\, _k(a s)≤ _kw_k\,m_k(s), with equality iff each ϕk _k peaks at the same a. Taking the max over a gives (20). In the disjoint-mode case, ϕk(a∣s) _k(a s) equals 1 exactly when a=aϕ⋆,ka=a_φ ,k and 0 otherwise, so ϕ¯(aϕ⋆,k∣s)=wk=1/K φ(a_φ ,k s)=w_k=1/K for each k and 0 elsewhere; the peak is 1/K1/K and the collapse error is 1−1/K1-1/K. □ Substituting into the chain. Under disjoint-mode population mixing, the accumulated non-mode probability ϵ¯HΦ(θ)≥(1−1/K)H ε_H (θ)≥(1-1/K)\,H is large, so Theorem 3.2’s sufficient condition for closeness to a single deterministic mode-user fails; the upper bound is vacuous, and the mixture environment cannot be reduced to one mode-user. The benefit instead is coverage: in Corollary 3.5, the geometric-concentration argument applies separately to each ϕk _k, but no single modal-exploit set Ax(k)A_x^(k) accumulates log-odds on more than a 1/K1/K fraction of steps, so the effective concentration rate slows roughly by a factor of 1/K1/K. The lemma is a per-turn guarantee conditional on pairwise-distinct modes; the informative-variation criterion (Remark B.10) is what keeps the buffer’s modes distinct during training. Coverage interpretation. The same effect shows up in behavior coverage. For a behavior set BxB_x that matters on task x, define qΦ(x)=∑kwkPrϕk[aϕ∈Bx∣x]q_ (x)= _kw_k _ _k[a^φ∈ B_x x]. A group of G rollouts from the population observes BxB_x with probability 1−(1−qΦ(x))G1-(1-q_ (x))^G. Population training helps when it raises this probability for the user behaviors that require strategies the modal one cannot serve. Behavior coverage and the per-turn floor in (21) are what carry the gain; raw model count by itself does not. Remark B.10 (Informative-variation criterion). For the moving target to remain useful, the simulator’s reward must keep the simulator from re-collapsing on a different mode. Purely adversarial rewards can collapse it toward refusal; purely cooperative ones can collapse it toward a trivial helper. Both destroy the simulator-side variance from Eq. 6 and leave the target stuck at a different fixed point. The curriculum reward in Appendix F.1 keeps the simulator in a regime where group rollouts still provide useful reward contrast. Appendix C Implementation Details C.1 Framework Training against a varied, updated pool of opponents is not restricted to fixed-API simulators. When both sides are trainable, Co-Training keeps the same diversity property while letting the opponent population update along with the agent. One pluggable opponent-generation function covers three paradigms: wrapping a fixed API gives frozen-population rotation; routing to a trainable SGLang engine gives self-play or Co-Training. 1. Online self-play (cf. SPIRAL [Liu et al. 2025c], SPICE [Liu et al. 2025d], Absolute Zero [Zhao et al. 2025b]): one model serves both roles with role-specific loss masks. 2. Co-Training (cf. Dr. MAS [Feng et al. 2026]): two separate models trained simultaneously on their respective turns of the same conversation. 3. Co-Training with opponent pool (ours): Co-Training augmented with a checkpoint pool P of historical opponent snapshots; each rollout loads opponent weights from a pool-sampled checkpoint, and GRPO’s clipped importance ratio corrects for the off-policy gap. Paradigm coverage versus other RL frameworks. Table 4 compares SCOPE’s paradigm coverage to other recent LLM RL frameworks. SCOPE adds no new RL primitive on top of the policy update. It composes the simulator-side paradigms our analysis needs (multi-turn dialogue rollouts, asymmetric two-agent rollouts, dual-model Co-Training, heterogeneous simulator rotation, historical-checkpoint pool sampling, and Verbalized Sampling at inference) into one pluggable interface on the SLIME training backend. The bottom three rows are the ones no other framework supports natively. Table 4: Training paradigms supported by LLM RL frameworks. ✓ = native support, ✗ = not supported, p = partial (e.g. available via patches but not first-class). SCOPE composes existing paradigms behind one interface; the differentiator is the bottom three rows (heterogeneous sim. rotation, checkpoint pool, and Verbalized Sampling), which no other framework supports natively. Property SCOPE (ours) SLIME verl Dr.MAS AstraFlow OpenRLHF Sotopia-RL Multi-turn dialogue (H≥10H≥ 10) ✓ ✗ p ✗ p ✗ ✓ Asymmetric two-agent ✓ ✗ ✗ ✓ p ✗ p Dual-model Co-Training ✓ ✗ ✗ ✓ ✓ ✗ ✗ Self-play ✓ p p ✓ ✓ ✗ ✗ Custom simulator reward ✓ ✗ ✗ p p ✗ ✗ Heterogeneous sim. rotation (K LLMs) ✓ ✗ ✗ ✗ ✗ ✗ ✗ Checkpoint pool (FIFO) ✓ ✗ ✗ ✗ ✗ ✗ ✗ Verbalized Sampling at inference ✓ ✗ ✗ ✗ ✗ ✗ ✗ Policy optimization. We use REINFORCE with group-relative normalization, adapted from GRPO [Shao et al. 2024] to the multi-turn setting. In the original bandit formulation of GRPO, the group consists of G parallel single-step responses to the same prompt. Here, each group contains G full dialogue trajectories from the same task, and the group-normalized advantage A^n=(R⁡(τn)−R¯)/σR A^n=(R(τ^n)- R)/ _R is the z-score of the terminal reward assigned uniformly to all agent tokens in trajectory n. For training stability we retain GRPO’s clipped importance-ratio surrogate [Schulman et al. 2017]: ℒ⁡(θ)=(st,atπ)∼ℬ[min⁡(ρt(θ)A^t,clip⁡(ρt(θ),1−ϵ,1+ϵ)A^t)],L(θ)=E_(s_t,a_t^π) [ ( _t(θ) A_t,\;clip ( _t(θ),1-ε,1+ε ) A_t ) ], (22) where ρt(θ)=πθ(atπ∣st)/πθold(atπ∣st) _t(θ)= _θ(a_t^π s_t)/ _ _old(a_t^π s_t) and ϵε is the clipping threshold. The clipping is inactive on-policy and reduces to pure REINFORCE when the policy has not drifted from the rollout policy. C.2 Training infrastructure Each iteration alternates a rollout stage and a training stage. The rollout stage runs a Gym-style loop [Brockman et al. 2016] per sample, alternating turns between the agent (local SGLang [Zheng et al. 2024] engine, log-probabilities preserved for the policy gradient) and a simulator assigned by the opponent-generation function (OpenAI-compatible API for frozen rotation, or a second SGLang engine for Co-Training and self-play; simulator tokens are masked from the policy loss). Each sample can draw a different simulator within the same batch. The training stage uses Megatron-LM [Shoeybi et al. 2019] (TP=4, P=1, BF16) with group-normalized advantages computed within each sample group (Eq. 22); for Co-Training, the two training groups run in parallel on disjoint GPU slices. Colocated dual-model layout. For Co-Training, agent and opponent share one 8×H100 node via a time-multiplexed schedule: parallel rollout, engine offload, parallel gradient steps on disjoint GPU slices, training offload, NCCL weight sync. A lightweight proxy layer remaps GPU offsets so each training group sees local ranks [0,3][0,3] regardless of physical placement. The whole stack ships as a ∼ 15-line external patch to Slime [Zhu et al. 2025], so it tracks upstream changes without a fork (in contrast to Dr. MAS [Feng et al. 2026], which forks veRL [Sheng et al. 2024]). AstraFlow [Zheng et al. 2026] addresses a different problem: it is a dataflow runtime for multi-policy agentic RL that handles scaling and scheduling across rollout, training, and dataflow. AstraFlow operates at the system level (for any multi-policy method); SCOPE operates at the method level (one pluggable opponent-generation interface for population Co-Training). The two are independent. C.3 Method comparison Table 5 summarizes how the seven training paradigms compared in the main results differ along three axes: the simulator pool, whether it is updated during training, and how the simulator is sampled per rollout. The fifth column states the property each method is designed to isolate. Table 5: Comparison of the seven training paradigms compared in the main results. “Updated?” indicates whether the simulator’s weights change during agent training. “Per-rollout sampling” specifies how the active simulator is selected for each rollout in a batch. Method Simulator pool Updated? Per-rollout sampling What it isolates 1. No training Base — — — — 2. Single-simulator (frozen) RL (Single) GPT-5-mini (single frozen LLM) No Single response Frozen single LLM Persona-Guided GPT-5-mini + persona prompt No One persona-conditioned response Prompt-only widening Verbalized Sampling GPT-5-mini No Sample one of k verbalized candidates Response-distribution diversity within one LLM 3. Multi-simulator (frozen) Ensemble Models (K=3K=3) Haiku 4.5, GPT-5-mini, Gemini 3 Flash No Cyclic rotation across rollouts Cross-family heterogeneity 4. Trainable simulator (ours) Co-Training 1 trainable LLM Yes Current weights ϕ(t)φ^(t) Simulator adaptivity Population Co-Training FIFO buffer of K=5K=5 historical ϕφ checkpoints Yes Uniform sample from buffer Diversity ++ adaptivity C.4 Hyperparameters All methods (RL Single, Verbalized Sampling, Ensemble, Co-Training, Co-Training with Population) share the optimizer and RL-loop settings in Table 6; only the opponent-generation mode and learning rate vary. Group Hyperparameter Value Optimizer Optimizer Adam (β1,β2)( _1, _2) (0.9,0.98)(0.9,0.98) Weight decay 0.10.1 Gradient clip (norm) 1.01.0 LR schedule constant RL loop Total training steps 250250 Prompts per rollout batch 1616 Samples per prompt (G) 88 Global batch size 128128 Rollout temperature 0.70.7 GRPO ϵlow _low / ϵhigh _high 0.20.2 / 0.280.28 KL loss coefficient β 0.0050.005 Entropy coefficient 0.00.0 Advantage normalization group-relative (within-prompt) Sequence lengths Max agent response length 32,76832,768 tokens Max context length 64,00064,000 tokens Max turns (P4G / τ2τ^2 / CooperBench) 1010 / 3030 / 5050 Precision BF16 Table 6: Shared hyperparameters used across all methods and tasks. Learning rate. All methods (Ensemble, Co-Training, Co-Training with Population) and Verbalized Sampling use 1×10−61×10^-6 across all tasks, tuned on the first 50 steps of τ2τ^2-bench Ensemble training (stable reward growth, entropy above 1.51.5 nats). Models. The trainable agents are Qwen3-4B-Instruct-2507 and Qwen3-8B for the dialogue tasks, and Qwen3.5-9B / Qwen3.5-27B for CooperBench [Qwen Team 2026]. All frozen simulators are accessed through OpenRouter using the slugs listed in Table 7. Three of them act as training simulators; all six are used at evaluation, with the three training models still in the panel for completeness. None of the evaluation-only models is used for checkpoint selection. Evaluator rollouts are deterministic at T=0T=0; training rollouts are stochastic at T=0.7T=0.7. Table 7: Closed-model simulators (OpenRouter slugs). Role indicates whether the model is used as a training simulator (Train), evaluation only (Eval), or both. Snapshot is the canonical model release on OpenRouter at access time. Family OpenRouter slug Snapshot Role OpenAI openai/gpt-5-mini 2025-08-07 Train + Eval Anthropic anthropic/claude-haiku-4.5 2025-10-01 Train + Eval Google google/gemini-3-flash-preview 2025-12-17 Train + Eval Z.ai z-ai/glm-5 2026-02-11 Eval MiniMax minimax/minimax-m2.7 2026-03-18 Eval DeepSeek deepseek/deepseek-chat-v3.1 2025-08-21 Eval Compute and memory. Table 8 reports per-step training compute and peak GPU memory relative to RL (Single), measured on the 8×H100 colocated layout (Appendix C.1). All methods share the training-step count of 250250; all headline comparisons in Tables 1 and 2 are at matched step count, not matched compute. Table 8: Per-method per-step training compute, peak GPU memory, and wall-clock per step. Wall-clock and total GPU-hour numbers are approximate, measured on 8×H100 with the Qwen3-4B agent on τ2τ^2-bench Retail; Qwen3-8B and Qwen3.5-9B CooperBench runs scale similarly. Qwen3.5-27B CooperBench runs use Tinker with LoRA adapters (Table 2 footnote) and are not directly comparable. Method Training compute Peak GPU memory Wall-clock / step RL (Single) 1.0×1.0× (ref) 1.0×1.0× ∼250 250 s Verbalized Sampling 1.0×1.0× (sim frozen) 1.0×1.0× ∼280 280 s Ensemble Models 1.0×1.0× (sim frozen) 1.0×1.0× ∼250 250 s Persona-Guided 1.0×1.0× (sim frozen) 1.0×1.0× ∼250 250 s Co-Training ∼2.0× 2.0× ∼2.0× 2.0× ∼500 500 s Population Co-Training ∼2.0× 2.0× ∼(1+K/2)× (1+K/2)× buffer ∼500 500 s The frozen-simulator methods (VS, Ensemble, Persona-Guided) do not back-propagate through the simulator, so per-step training compute matches RL (Single). VS decodes a verbalized KVS=5K_VS=5-candidate distribution per simulator turn, which raises rollout time but not training FLOPs. Co-Training and Population Co-Training back-propagate through both sides on the same rollout, roughly doubling per-step training compute. Population Co-Training holds a FIFO buffer of K=5K=5 recent simulator checkpoints; only one is the active opponent at any step, so per-step training compute equals Co-Training while cumulative parameter memory grows with K. Total GPU-hours per 250-step run on τ2τ^2-bench Retail (Qwen3-4B agent, 8×H100): ∼280 280 for RL (Single), ∼560 560 for Co-Training, ∼560 560 for Population Co-Training. Our comparison isolates the training-environment intervention at matched step count; an iso-compute sweep is left to future work. The Qwen3.5-27B CooperBench runs use Tinker’s API training with LoRA adapters instead of full-parameter Megatron training, so the per-step wall-clock and GPU-hour numbers above do not apply to those runs. Appendix D Prompts and Templates This appendix lists the full system-prompt templates used for each benchmark. Curly braces (e.g. persona_text, word_limit) are Python format-string placeholders filled in per-rollout from the corresponding data source. P4G Persuader (agent) system prompt You are an Amazon Mechanical Turk worker completing a $2 communication task. • You are motivated by this task payment — you value every cent you earn. • Act naturally as the person in the <persona> tag — think and respond as they would, including their quirks, beliefs, biases, and reasoning. • Complete the communication task outlined in the <scenario> tag as the persona would naturally respond. • Respond in a real-time chat interface. Keep each response under word_limit words, conversational, and authentic — avoid formal, robotic, or repetitive language. • Guide the conversation for num_turns / 2 chat turns. Do not propose or agree to a donation too early — build rapport gradually and naturally. Only output your reply to your chat partner — do not explain your reasoning. persona_text <scenario> You are randomly assigned to be the Persuader in this communication task. Your job is to persuade your partner to donate some or all of their incoming $2.00 task payment to a children’s charity called Save the Children. You will earn a bonus equal to whatever amount your partner donates if they agree to donate. Required actions: ask for a specific donation amount; explain that the donation will be deducted from the task payment; if asked, explain that the research team will collect and forward all donations to the charity. Strategy: start with a greeting or small talk; build rapport before discussing donations; mention the minimum-turn requirement only if necessary to avoid trivial back-and-forth. </scenario> <rules> Responses can sometimes contain minor grammatical errors to sound natural. Begin with a brief greeting. Stay under word_limit words. </rules> Figure 5: Persuader (agent) system prompt for Persuasion for Good. The persona block is loaded per-rollout from the convokit P4G corpus [Wang et al. 2019]. P4G Persuadee (user simulator) system prompt You are an Amazon Mechanical Turk worker completing a $2 communication task. • You are motivated by this task payment — you value every cent you earn. • Act naturally as the person in the <persona> tag. • Respond in a real-time chat interface, under word_limit words, conversational and authentic. Only output your reply to your chat partner — do not explain your reasoning. persona_text <scenario> You will now start a conversation with your partner about a children’s charity. Please don’t game the task with short or meaningless replies. </scenario> <task> You have just made $2.00. You have access to this money whenever you like, and you are free to use it however you wish. You are about to interact with an individual about a children’s charity. They may try to get you to donate. Make the decision wisely; do not give the money away easily. </task> <rules> • Responses can sometimes contain minor grammatical errors to sound natural. • To donate or give money, say [DONATE $N] or [GIVE $N] where N is the amount (integer or decimal). Without this marker, the donation is not registered. • Stay under word_limit words. </rules> Figure 6: Persuadee (user simulator) system prompt for Persuasion for Good. The structured [DONATE $N] marker is parsed by the reward function [Wang et al. 2019]. Verbalized Sampling instruction (appended to persuadee prompt) ## Response Diversity Instructions Instead of committing to a single response, generate n plausible replies you (as the persuadee) might naturally give to your chat partner, along with an estimated probability for each. The replies should be genuinely diverse — they should differ in phrasing, tone, skepticism level, and (where appropriate) whether or not to donate. IMPORTANT: if a candidate reply represents a decision to donate or give money, that candidate must include the literal structured marker [DONATE $N] or [GIVE $N] where N is the amount. Otherwise the donation will not be registered. Candidates that do not involve donating should omit the marker entirely. Return ONLY a JSON object with the key "responses" (a list of objects with ’text’ and ’probability’): • ’text’: the reply string only, under word_limit words, no explanation. • ’probability’: a number in [0,1][0,1] representing how likely you are to give this particular reply (probabilities should roughly sum to 1.01.0). Example format: "responses": ["text": "I’m not sure, I earned this money and want to spend it on myself.", "probability": 0.5, "text": "That’s a worthy cause. [GIVE $0.25]", "probability": 0.2, "text": "Can you tell me more first?", "probability": 0.3] Output ONLY the JSON object: no markdown fences, no explanations, no extra text. Figure 7: Verbalized-sampling instruction block appended to the persuadee system prompt for the VS baseline [Zhang et al. 2025a]. At rollout time, one of the n candidate replies is sampled with the verbalized probabilities. τ2τ^2-bench user simulator guidelines # User Simulation Guidelines You are playing the role of a customer contacting a customer service representative. Your goal is to simulate realistic customer interactions while following specific scenario instructions. ## Core Principles • Generate one message at a time, maintaining natural conversation flow. • Strictly follow the scenario instructions you have received. • Never make up or hallucinate information not provided in the scenario instructions; treat anything not provided as unknown or unavailable. • Avoid repeating the exact instructions verbatim. Use paraphrasing and natural language. • Disclose information progressively. Wait for the agent to ask for specific information before providing it. ## Task Completion • Continue the conversation until the task is complete. • If the instruction goal is satisfied, generate the ###STOP### token to end the conversation. • If transferred to another agent, generate the ###TRANSFER### token. • If the scenario does not provide enough information to continue, generate the ###OUT-OF-SCOPE### token. The goal is realistic, natural conversations while strictly adhering to the provided instructions and maintaining character consistency. <scenario> instructions </scenario> Figure 8: τ2τ^2-bench user-simulator system prompt. The instructions placeholder is filled per-task with the scenario-specific instruction block from the retail or airline split [Barres et al. 2025]. CooperBench coding-agent system prompt You are a helpful assistant that can interact with a computer. You are agent_id working as a team with: agents_str. You are all working on related features in the same codebase. Each agent has their own workspace. Use the send_message tool to coordinate with teammates. Figure 9: Coding-agent system prompt for CooperBench (cooperative setting) [Khatua et al. 2026]. agent_id and agents_str are filled per-rollout with the agent’s id and the comma-separated teammate list. The solo and baseline settings drop the team coordination block. Appendix E Human Study Our main results use LLM evaluators that share RLHF biases with the training simulators [Zhou et al. 2026]. The human study tests whether Co-Training’s gains hold up against real users on both τ2τ^2-bench (task-oriented) and Persuasion for Good (open-ended), following Zhou et al. 2026’s τ2τ^2 protocol and Wang et al. 2019’s original P4G setup. Recruitment is on Prolific; the study runs on a Cloudflare-tunneled chat interface, described next. Interface. A Prolific URL mints a fresh session and renders the task instruction in a structured right panel (goal, role, conditional behaviors, plus a Travelers callout when the τ2τ^2 scenario involves multiple passengers) next to a single-message-per-turn streaming chat (Figures 10, 11). The End button (matching /stop from Zhou et al. 2026) opens a confirmation modal that warns the participant to verify each requested change first, then swaps the instruction panel for an inline survey (Figures 12, 13). On submission, the participant auto-redirects to Prolific with a manual code copy as fallback (Figure 14). Recruitment. Three Prolific screeners filter participants: English as first language, prior approval rate ≥95%≥ 95\%, prior submissions ≥50≥ 50. The study is restricted to six English-native countries (US, UK, Canada, Ireland, Australia, New Zealand) and to desktop devices, with the balanced-sample-on-Sex quota targeting a 50/5050/50 split. Expected demographics span ages 18–75 (median ∼ 35), skewed toward part-time and full-time employment. Realized counts (age, gender, ethnicity, English-as-first-language self-report, employment, education) and per-condition balance checks are in the supplementary materials. Compensation and IRB. Each participant is paid $3.17 for an estimated 10-minute session ($19.02/hr, within Prolific’s competitive band). Engagement bonuses depend on conversation length and survey completeness only, never on task success or donation amount, so outcome measures stay unbiased. If a session is interrupted by an infrastructure fault (server error, model timeout, broken redirect), the participant gets the full study reward regardless of completion. The protocol has IRB approval at the authors’ institution. Survey instruments, recruitment materials, and anonymized response data are in the supplementary materials. Figure 10: τ2τ^2-bench landing view. The right panel renders the structured task profile (goal, role information, conditional behaviors, style notes, and a sky-blue Travelers callout when the scenario involves multiple passengers). The left panel is a single-message-per-turn streaming chat with the agent, with an explicit End button next to the input. Light-mode rendering for legibility; the deployed interface ships in dark mode. Figure 11: Persuasion for Good landing view. The right panel renders the persuadee role-play prompt drawn from the ConvoKit corpus, including the persona’s demographics, salient personality traits, and behavioural guidance (e.g., “Don’t agree or refuse on the first message”). The chat panel hosts the open-ended donation conversation; the agent does not invoke tools on this task. Figure 12: τ2τ^2-bench survey panel. Rendered after the participant confirms the End modal. The eight 1–7 Likert items (task success, helpfulness, honesty, efficiency, instruction-following, safety, frustration, overall satisfaction) replace the instruction panel on the right; the chat history is retained on the left, now disabled, so the participant can refer to specific exchanges while answering. A free-text field at the bottom captures qualitative feedback; Submit is gated until every Likert row has a value. Figure 13: Persuasion for Good survey panel. Differs from the τ2τ^2 survey in two ways: the first item is a continuous donation amount in [$0,$2][\$0,\$2] rather than a Likert score, and the five Likerts measure perceptions of the persuader (argument quality, empathy, perceived manipulation, engagement, future donation likelihood) rather than task-execution dimensions. Figure 14: Debrief page. Surfaces the Prolific completion code, an eight-second auto-redirect countdown back to Prolific, the post-hoc task-outcome verdict from the τ2τ^2 evaluator, and a debrief block revealing which agent condition the participant interacted with (blinded during the chat itself). Conditions. For each task we evaluate four policies trained on Qwen3-4B-Instruct: Base (untrained), RL (Single) (trained against GPT-5-mini), + Verbalized Sampling, and + Co-Training (Section 3.4). Assignment is server-side, stratified by least-populated slot with hash-based tie-breaking on the Prolific ID, which keeps the four cells within ±1± 1 session of each other. Task pools. The τ2τ^2 pool is the released benchmark from Barres et al. 2025: 15 retail and 15 airline scenarios, assigned round-robin. The agent dispatches tool calls through a per-session τ2τ^2 runtime mirroring the benchmark database and policy. The P4G pool is built from Wang et al. 2019’s ConvoKit corpus (1,285 speakers): we keep the 592 who appear in the persuadee role, sample 30 with a fixed seed, and render each as a role-play prompt with demographics (age, sex, race, marital status), education, employment, religion, political ideology, and any salient Big-Five trait. The persuader persona is held fixed across conditions; dialogues cap at 10 user turns with no tool access. Surveys. The τ2τ^2 survey, adapted from Zhou et al. 2026, has eight 1–7 Likert items: task success (“Did the agent complete your request?”), helpfulness, honesty (“no hallucinations”), efficiency, instruction-following, safety, frustration (reverse-coded), and overall satisfaction. The objective task reward is computed post-hoc by the τ2τ^2 evaluator on the recorded transcript, falling in [0,1][0,1] as a partial-credit aggregate of action-match, database-state, and information-communication subscores. The P4G survey, following Wang et al. 2019, has one continuous intended donation in USD ∈[0,2]∈[0,2] and five 1–7 Likert items: argument quality, empathy, manipulation (reverse-coded), engagement, and future donation likelihood. Both surveys close with an optional free-text field for examples and suggestions. Sample size and tests. Each (condition, task) cell uses N=40N=40, for 4×2×40=3204× 2× 40=320 total Prolific sessions. The cell size detects Cohen’s d≈0.55d≈ 0.55 on Likert outcomes at 80%80\% power (α=0.05α=0.05, Welch’s t). For donations, this corresponds to a detectable lift of $0.30 at the empirical donation standard deviation of ∼$0.65 \$0.65. Per-condition means are reported with 95%95\% bootstrap percentile confidence intervals (10,000 resamples). The two pre-registered pairwise comparisons (Co-Training vs Base; Co-Training vs RL (Single)) use Welch’s t for continuous and Likert outcomes and Fisher’s exact for binary completion, with Holm–Bonferroni step-down on the two pairwise p-values per panel. Adversarial stratification. The τ2τ^2 analysis was pre-registered with a split on airline tasks 4, 6, 9, 10, where the participant is instructed to push back, supply false information, or attempt jailbreaks. On these adversarial scenarios we predicted brittle conciliation: RL (Single), trained only against GPT-5-mini, disengages politely on user pushback rather than persisting toward the goal. Co-Training, which sees a population that includes adversarial behaviors during training, recovers on this subgroup; the subgroup cells (N≈12N≈ 12) are reported with explicit power caveats. Derived measures. Beyond survey items, we extract three behavioral measures from the recorded transcripts: P4G conversation length (turns, capped at 10), P4G donation-ask count, and a τ2τ^2 goal-abandonment indicator. These probe the mechanism of any failure rather than its surface signal. The pre-registration (primary outcomes: τ2τ^2 task reward and P4G intended donation; secondary outcomes: full surveys plus the three derived measures; corrections as above) was filed before recruitment. Results. Figures 15 and 16 report the human-study results across the four conditions on both benchmarks. Co-Training is the top method on τ2τ^2-bench task outcome and the Likert quality metrics, while Verbalized Sampling is the top method on P4G intended donation. Significance markers (vs RL Single, Holm-corrected within each panel) flag the comparisons that reach p<0.05p<0.05 at N=40N=40. Figure 15: Human study on τ2τ^2-bench. Each subplot reports mean ±1± 1 std across N=40N=40 Prolific participants per condition. ∗p<0.05^*\,p<0.05, p∗⁣∗<0.01^**\,p<0.01 vs RL (Single) (Welch’s t, Holm–Bonferroni within each panel; 2 pre-registered pairwise tests). Figure 16: Human study on Persuasion for Good. Each subplot reports mean ±1± 1 std across N=40N=40 Prolific participants per condition. ∗p<0.05^*\,p<0.05, p∗⁣∗<0.01^**\,p<0.01 vs RL (Single) (Welch’s t, Holm–Bonferroni within each panel; 2 pre-registered pairwise tests). Appendix F Extended Experiments F.1 Where Population Co-Training’s Gain Comes From Population Co-Training was the top method on every benchmark in §4.2. Two design hypotheses follow from the theory: the buffer must be wide enough to preserve real cross-checkpoint disagreement (Q3a, pool size K), and the simulator reward must keep each checkpoint informative rather than re-collapsed on a new mode (Q3b, reward design). We probe each on τ2τ^2-bench Retail. The buffer must be wide, but not stale. Q3a asks how many checkpoints the population should hold. Sweeping K∈1,3,5,10K∈\1,3,5,10\ with one checkpoint every four training steps (Figure 17), K=1K=1 reduces to a single moving target. Larger K mixes in older checkpoints whose simulator capability lags the current one; these stale partners dilute the gradient signal. K=5K=5 and K=10K=10 perform comparably at the top, with K=5K=5 slightly ahead on both P4G and τ2τ^2-Retail; K=1K=1 and K=3K=3 fall meaningfully behind. The pool-size benefit plateaus around K=5K=5: the pool’s value comes from preserving current disagreement, so stale checkpoints hurt at the same rate that fresh ones help. Figure 17: K=5K=5 peaks above both extremes; the optimum is interior because stale checkpoints dilute the population. Eval reward over training for K∈1,3,5,10K∈\1,3,5,10\ on P4G (left) and τ2τ^2-Retail (right); checkpoint cadence is every four training steps. Asymptote ordering: K=5>K=10>K=3>K=1K=5>K=10>K=3>K=1. The simulator reward must preserve variation. For Q3b we ablate the simulator’s reward design among three variants: adversarial (rϕ=−rπr_φ=-r_π), cooperative (rϕ=rπr_φ=r_π), and a SPICE-style curriculum (rϕr_φ peaks at a target within-batch variance). The two endpoints both collapse the simulator onto a new dominant mode and undo the moving target; only the curriculum reward preserves within-batch variation (Remark B.10) and yields gains over the no-co-training K=3K=3 ensemble baseline. Full variants, numbers, and training curves are in Appendix F.8. F.2 Reward-quadrant ablation Our main-text results in Section 4 pair each task with the reward structure that best matches its role-and-objective configuration. Persuasion for Good uses the donation-based adversarial reward that the task definition already encodes. τ2τ^2-bench uses the curriculum reward studied in Appendix F.1, which shapes the simulator toward the within-batch variance regime of Remark B.10. CooperBench is symmetric: both sides share the binary task-success reward shipped with the benchmark, which is the cooperative case. This appendix ablates the pairing. For each cell of the asymmetric/symmetric × adversarial/cooperative spectrum, we swap in a reward drawn from a different quadrant and rerun the same method set, to test whether the main-text conclusions depend on the default reward or on population training itself. Figure 18: τ2τ^2-bench with Qwen3-8B. Scaling the trainable policy from Qwen3-4B-Instruct to Qwen3-8B preserves the ordering from Figure 4: RL (Single) collapses below the untrained baseline, every population-based method improves steadily, and Population Co-Training is the top curve. Noise reflects the smaller number of eval episodes in this setting. F.3 Empirical training dynamics across methods Figure 19 extends Figure 3 with additional batch-level diagnostics. RL against a single frozen simulator collapses on every diagnostic the theory predicts. Zero-variance batches climb from 60%60\% to over 85%85\% (Lemma 3.3). Policy entropy drops from 1.91.9 to 0.40.4 nats (Corollary 3.5). All-failure batches rise to 70%70\% as the policy concentrates on a mode-exploit strategy that wins against no one. Within-simulator Verbalized Sampling slows the collapse but does not stop it. Cross-family ensembles (K=3K=3) and the two Co-Training variants are the only methods that hold all four diagnostics healthy throughout training. Figure 19: Empirical training dynamics on τ2τ^2-bench Retail (full time series). Zero-variance batch fraction (top-left), all-success (top-right), all-failure (bottom-left), and policy entropy (bottom-right). Single-simulator RL (blue) blows up on every panel; Co-Training and Population Co-Training are the only methods that keep all four healthy. Shaded bands are ±1σ± 1σ over three seeds. F.4 Per-benchmark training, eval, and entropy curves across all settings We report the full training-time picture for every method on each benchmark, with the same three signals tracked in §3.3 (training reward, OOD eval reward, policy entropy). Three seeds per method; ±1σ± 1σ shading on the OOD panel. Each benchmark uses its own scale and method set. P4G and τ2τ^2-bench Retail (Figures 20, 21) show the same qualitative pattern. RL (Single) takes the highest training reward against its training simulator and the lowest policy entropy, but its OOD eval slides back toward the untrained baseline. Verbalized Sampling and frozen ensembles trade a noisier, lower training reward for a higher steady OOD eval; Co-Training and Population Co-Training extend the OOD gain further and keep entropy in the 0.80.8–1.21.2 nat range throughout training. CooperBench (Appendix Figures 23 and 24) adds a conversation-turn diagnostic. Cross-play against a fixed partner produces a turn-count curve that rises and then drops: once the policy finds a short strategy the frozen partner accepts, conversations get shorter. Self-play and Population Co-Training do not show this drop. The partner keeps adapting, so longer multi-turn solutions stay rewarded and the turn count keeps climbing. The 9B and 27B runs show the same qualitative pattern, with the 9B curves visibly noisier reflecting the smaller model’s higher per-step variance on SWE-style tasks. Collapse persists at the 8B scale. Scaling the trainable policy from Qwen3-4B-Instruct to Qwen3-8B does not eliminate simulator collapse: RL (Single) still saturates its training reward, peaks transiently on OOD, and crashes its policy entropy (Figure 22). The 8B run appears to collapse later than the 4B run, though we do not measure onset precisely; the qualitative pattern is the same. Figure 20: Persuasion for Good: training, OOD eval, and policy entropy. Six methods, three seeds each; ±1σ± 1σ shading on OOD. Training reward against the GPT-5-mini training simulator (a) is highest for RL (Single), which corresponds to the lowest agent entropy (c) and the worst OOD eval (b). Figure 21: τ2τ^2-bench Retail: training, OOD eval, and policy entropy. Seven methods, three seeds each. Same qualitative pattern as Figure 20: RL (Single) wins training reward and loses OOD; Co-Training and Population Co-Training are best on the held-out panel. Figure 22: τ2τ^2-bench Retail (Qwen3-8B). Six methods, three seeds each; ±1σ± 1σ shading on OOD. The collapse pattern from Figure 21 appears at the 8B scale, qualitatively similar but visually later in training: RL (Single)’s training reward saturates, OOD eval peaks and slides, and policy entropy crashes. The per-step curves are also noisier across all three panels; the bigger policy has wider per-step variance at matched eval-episode counts. Figure 23: CooperBench (Qwen3.5-9B): training, eval, entropy, and conversation turns. The 9B run shows the same overfit signature as Figure 24 (Qwen3.5-27B) but with visibly more per-step variance, characteristic of SWE-style tasks at the smaller scale. Cross-play against a fixed Haiku partner or against a K=3K=3 frozen ensemble plateaus and starts dropping in conversation turns as a short exploit strategy takes over; Self-play and Population Co-Training keep climbing. Figure 24: CooperBench (Qwen3.5-27B): training, eval, entropy, and conversation turns. Cross-play against a fixed Haiku partner or against a K=3K=3 frozen ensemble shows the overfit signature on conversation turns: turns rise early as the policy learns to interact, then drop as a short exploit strategy takes over. Self-play and Population Co-Training avoid this regression; turns keep climbing as the partner co-evolves. F.5 Proof of Proposition 3.7 This subsection proves Proposition 3.7 by the same coupling argument as Theorem 3.2, then states a γ-sharpening corollary on tail-behavior coverage. Proof of the gradient bound. Couple the trajectory τVS _VS in MVSM_VS (simulator samples from pϕVSp^VS_φ) and the trajectory τref _ref in MrefM_ref (simulator samples from P) using shared agent randomness. At each simulator turn, given a matched prefix, sample (atVS,atref)(a_t^VS,a_t^ref) from the maximal coupling between pϕVS(⋅∣st,atπ)p^VS_φ(· s_t,a_t^π) and P(⋅∣st,atπ)P(· s_t,a_t^π), which disagrees with probability at most η⁡(st,atπ)η(s_t,a_t^π) by hypothesis. By the union bound over simulator turns, Pr[τVS≠τref]≤[∑t=1Hη(st,atπ)]=η¯H(θ), [ _VS≠ _ref]\;≤\;E\! [ _t=1^Hη(s_t,a_t^π) ]\;=\; η_H(θ), which gives DTV(PVSθ,Prefθ)≤η¯H(θ)D_TV(P^θ_VS,P^θ_ref)≤ η_H(θ). The gradient bound then follows by the bounded-integrand TV inequality (Appendix B.2) applied to f⁡(τ)=R⁡(τ)Sθ(τ)f(τ)=R(τ)S_θ(τ) with ‖f‖≤RmaxB\|f\|≤ R_ B, exactly as in the proof of Theorem 3.2. □ γ-sharpening exponentially suppresses tail behaviors. Combined with γ-sharpening (Appendix B.3), the recovery assumption separates VS from direct prompting on tail behaviors: direct prompting suppresses them exponentially in γ, while VS preserves them up to η. Proposition F.1 (γ-sharpening tail suppression). Fix a state s and let b⋆=arg⁡maxb⁡P⁡(b∣s)b = _bP(b s) with mass m=P⁡(b⋆∣s)m=P(b s). Let B⊂ℬB be a set of nonmodal behaviors with reference mass P⁡(B∣s)=ρP(B s)=ρ and maximum per-behavior mass maxb∈B⁡P⁡(b∣s)≤λm _b∈ BP(b s)≤λ m for some λ∈[0,1)λ∈[0,1). Under direct γ-sharpened prompting Pγ(b∣s)∝P(b∣s)γP_γ(b s) P(b s)^γ with γ>1γ>1 (Proposition B.2), Pγ(B∣s)≤ρmλγ−1.P_γ(B s)\;≤\; ρm\,λ^γ-1. Under VS with reference-recovery error η, pϕVS(B∣s)≥ρ−η.p^VS_φ(B s)\;≥\;ρ-η. Proof. ∑b∈BP(b)γ=∑b∈BP⁡(b)⋅P(b)γ−1≤ρ(λm)γ−1 _b∈ BP(b)^γ= _b∈ BP(b)· P(b)^γ-1≤ρ(λ m)^γ-1 using P⁡(b)≤λmP(b)≤λ m on B. The denominator of PγP_γ is at least the modal contribution mγm^γ, giving Pγ(B)≤ρ(λm)γ−1/mγ=(ρ/m)λγ−1P_γ(B)≤ρ(λ m)^γ-1/m^γ=(ρ/m)\,λ^γ-1. For the VS bound, |pϕVS(B)−P⁡(B)|≤DTV(pϕVS,P)≤η|p^VS_φ(B)-P(B)|≤ D_TV(p^VS_φ,P)≤η. □ ∎ For typical empirical estimates γ∈[6,66]γ∈[6,66] (Appendix B.3) and λ bounded away from 11, Pγ(B)P_γ(B) collapses to near zero while pϕVS(B)p^VS_φ(B) stays within η of ρ. The RL consequence is Proposition 3.7: the policy gradient under VS approximates the reference-user gradient, with the gap controlled by η¯H η_H. Honest accounting of assumptions. The reference-recovery assumption DTV(pϕVS,P)≤ηD_TV(p^VS_φ,P)≤η is an empirical claim. It can fail if the simulator’s verbalized output systematically misses behavior types (for example, the K candidates always come from the cooperative half of the response distribution). The most direct test would sample the simulator under VS and cluster responses into behavior types, then compare the empirical distribution to a behavior-level reference; we report transcript inspections in Appendix F.9 but defer quantitative behavior-coverage measurements to future work. Reference recovery is necessary but not sufficient: P must also place mass on behaviors the policy needs to transfer; Proposition F.1 formalizes one such asymmetry. The real-user step requires an additional assumption DTV(P,Preal)≤κD_TV(P,P_real)≤κ that VS itself does not establish; the human study in Appendix E is the test of that step. F.6 Verbalized Sampling mitigates simulator collapse We isolate the effect of Verbalized Sampling [Zhang et al. 2025a] on the same training simulator used in our main experiments (GPT-5-mini). Figure 25 compares RL (Single) and RL (Single) ++ Verbalized Sampling under matched hyperparameters and three seeds each. Without VS, the policy follows the simulator-collapse signature established in §3.3: training reward climbs cleanly while OOD eval peaks and slides back, and policy entropy crashes to near zero. With VS the simulator is queried for a verbalized response distribution and the rollout is drawn from it, which restores enough simulator-side variance (Lemma 3.3) to slow the geometric concentration of Corollary 3.5. Figure 25: Verbalized Sampling mitigates simulator collapse against GPT-5-mini. Three seeds per setting. (a) Training reward: both runs climb, with ++VS reaching a slightly higher plateau because the gradient signal is preserved on more batches. (b) OOD eval reward on the held-out 6-model panel (±1σ± 1σ shaded): ++VS peaks much higher and degrades far less than RL (Single), narrowing the gap to the untrained baseline. (c) Policy entropy: RL (Single) collapses to near zero, while ++VS holds entropy near 0.70.7–0.80.8 nats throughout training. VS reduces but does not eliminate simulator collapse; the residual gap motivates the Co-Training experiments in §4.2. F.7 Verbalized Sampling on larger models In this part we show training dynamics against the larger GPT-5 simulator. Given the limited budget constraint and fair comparison in Table 1, we use GPT-5-mini as the simulator across all settings in the main experiments. The original VS paper notes that smaller models suffer from “cognitive overload” when asked to verbalize a distribution while solving the task; against GPT-5 this is less of a concern, so VS’s mitigation effect is more pronounced. Figure 26 shows the same three diagnostics against GPT-5. The collapse signature persists but is delayed: GPT-5 is less modal than GPT-5-mini, so the training reward climbs more slowly and tops out around 0.60.6–0.80.8, and the entropy collapse for RL (Single) sets in much later than in the GPT-5-mini setting. OOD eval for RL (Single) still peaks at ≈0.65≈ 0.65–0.670.67 and then degrades. With Verbalized Sampling the training reward is noisier and lower, but OOD climbs steadily and ends near 0.700.70, and entropy stays well above zero with substantial variance throughout. The qualitative story matches the GPT-5-mini ablation in §F.6: VS reduces, but does not eliminate, simulator collapse. Figure 26: Verbalized Sampling against GPT-5. Three seeds per setting. (a) Training reward: RL (Single) climbs to a plateau in the 0.60.6–0.80.8 range; ++VS is noisier and lower because the K-modal verbalized simulator gives less consistent reward signal. (b) OOD eval (held-out 6-model panel, ±1σ± 1σ shaded): RL (Single) peaks near 0.660.66 and degrades; ++VS climbs more slowly but ends near 0.700.70. (c) Policy entropy: RL (Single) stays high for most of training and then crashes; ++VS holds entropy above 11 nat throughout with substantial fluctuation. F.8 Simulator-reward ablation Q3b asks whether any choice of simulator reward yields the moving target, or whether the reward must be carefully shaped. We compare three simulator-reward variants (Table 9) against the no-co-training K=3K=3 ensemble baseline (Figure 27). The two endpoints break the moving target, in mirrored ways. An adversarial reward (rϕ=−rπr_φ=-r_π) collapses the simulator to ∼98% 98\% refusal and drops eval reward to 0.070.07. A cooperative reward (rϕ=rπr_φ=r_π) pushes pushback to ∼2% 2\%, letting the policy reward-hack a trivial helper while eval reward drops from 0.270.27 to 0.170.17. Only the curriculum reward keeps opponent reward near 0.450.45, the regime of maximum within-batch variance (Remark B.10), and reaches ∼0.40 0.40 eval. Both extremes collapse the simulator onto a new dominant mode, and once the mode is fixed again the simulator-collapse chain rebinds: the moving target stops moving. Population Co-Training helps only when the simulator reward preserves variation across checkpoints. Table 9: Simulator-reward variants. rπr_π is the policy’s per-rollout reward; σπ2σ^2_π its within-group variance. Curriculum follows SPICE-style variance shaping [Liu et al. 2025d]. Variant Simulator reward rϕr_φ Adversarial −rπ-r_π Cooperative rπr_π Curriculum exp⁡(−(σπ2−0.25)20.02) \! (- (σ^2_π-0.25)^20.02 ) Figure 27: Co-Training requires a carefully chosen simulator reward (τ2τ^2-bench Retail). Top-left: agent training reward; cooperative reward (green) reaches the highest training reward by reward-hacking a trivially helpful simulator. Top-right: opponent reward; only the curriculum reward (blue) stays balanced near 0.450.45. Bottom-left: agent policy entropy; both extremes collapse to 0.010.01–0.040.04. Bottom-right: held-out eval reward; only the curriculum reward beats the no-co-training ensemble baseline. F.9 Within-batch rollouts: from diverse openings to a single strategy We illustrate the policy-side signature of simulator collapse by inspecting three within-batch rollouts from the same starting context at three training stages (early / mid / late). Early in training the agent samples varied strategies and the simulator responds with varied content; by mid-training the rollouts begin to share scaffolding; by late training all three rollouts within a batch are nearly word-for-word the same, which is what the geometric strategy-mass concentration of Corollary 3.5 looks like in transcript space. F.10 Training on more models We replicate the main P4G and τ2τ^2-bench Retail experiments with Olmo-3-7B-Instruct [Olmo et al. 2025] as the trainable agent, keeping every other element of the setup unchanged. Two differences relative to the Qwen3-4B-Instruct runs are notable. First, Olmo-3-7B-Instruct starts with higher base policy entropy than Qwen3-4B-Instruct (≈1.85≈ 1.85 vs ≈1.55≈ 1.55 nats); its RLHF profile is less sharpened. Second, its Base task performance is lower (0.340.34 vs 0.400.40 on τ2τ^2-Retail, 0.350.35 vs 0.430.43 on P4G), consistent with its smaller post-training budget. Despite these starting-point differences, the qualitative training dynamics match the Qwen3-4B-Instruct results: RL (Single) saturates training reward, peaks transiently on OOD eval, and crashes policy entropy from the higher initial point. Both Verbalized Sampling and Co-Training recover most of the OOD gap and preserve entropy, and Population Co-Training is the strongest method on both benchmarks. Figures 28 and 29 report the full curves. Figure 28: Persuasion for Good (Olmo-3-7B-Instruct). Three panels: training reward, OOD eval reward, policy entropy. The qualitative simulator-collapse signature reproduces: RL (Single) saturates training reward, peaks transiently on OOD, and crashes its (higher initial) entropy; Verbalized Sampling, Ensemble, Co-Training, and Population Co-Training preserve entropy and close most of the held-out gap. Figure 29: τ2τ^2-bench Retail (Olmo-3-7B-Instruct). Same three panels as Figure 28. Olmo-3-7B-Instruct’s Base success rate is lower than Qwen3-4B-Instruct’s (0.340.34 vs 0.400.40), but the collapse-and-recovery pattern is the same.

Instant research discovery