Paper deep dive
Direct Diversity Optimization for Diverse Successful Trajectories in Preference Post-Training
Junwon Ko, Dong-Jae Lee, Minchan Kwon, Sunghyun Baek, Junmo Kim
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 96%
Last extracted: 9/10/2026, 3:39:33 AM
Summary
The paper introduces Direct Diversity Optimization (DDO), an offline post-training method for Large Language Model (LLM) agents in sequential decision tasks. DDO addresses the issue where standard preference optimization narrows the output distribution to a single successful trajectory. It combines Divergence-Tree Collection (DTC), which constructs state-aligned branch sets of successful and failed alternatives, with the Reference-Relative Target-Odds Objective (RTO), which trains the model to match reference-relative targets over successful branches. DDO demonstrates superior task success and successful strategy coverage compared to baselines like DPO and DivPO on benchmarks including BabyAI, BabaIsAI, and WebShop.
Entities (8)
Relation Signals (6)
Direct Diversity Optimization → evaluatedon → BabyAI
confidence 100% · DDO achieves the strongest task success and successful strategy coverage among the compared post-training methods across BabyAI, BabaIsAI, and WebShop.
Direct Diversity Optimization → evaluatedon → BabaIsAI
confidence 100% · DDO achieves the strongest task success and successful strategy coverage among the compared post-training methods across BabyAI, BabaIsAI, and WebShop.
Direct Diversity Optimization → evaluatedon → WebShop
confidence 100% · DDO achieves the strongest task success and successful strategy coverage among the compared post-training methods across BabyAI, BabaIsAI, and WebShop.
Direct Diversity Optimization → usescomponent → Divergence-Tree Collection
confidence 95% · DDO, an offline post-training method that combines Divergence-Tree Collection (DTC) with the Reference-Relative Target-Odds Objective (RTO).
Direct Diversity Optimization → usescomponent → Reference-Relative Target-Odds Objective
confidence 95% · DDO, an offline post-training method that combines Divergence-Tree Collection (DTC) with the Reference-Relative Target-Odds Objective (RTO).
Direct Diversity Optimization → outperforms → DPO
confidence 90% · DDO achieves the strongest task success and successful strategy coverage among the compared post-training methods... DDO also achieves higher task success and coverage than successful-only imitation and decoding-time diversification controls.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:LLM agents for sequential decision tasks are often post-trained with trajectory-level outcome labels, but such labels provide little supervision for preserving multiple successful branches from the same decision state. We study this problem as successful strategy coverage: how broadly a model realizes distinct successful strategies under a fixed rollout budget. We present Direct Diversity Optimization (DDO), an offline post-training method that combines Divergence-Tree Collection (DTC) with the Reference-Relative Target-Odds Objective (RTO). DTC constructs state-aligned branch sets rooted at shared decision states, and RTO trains the model to match reference-relative targets over successful alternatives. DDO achieves the strongest task success and successful strategy coverage among the compared post-training methods across BabyAI, BabaIsAI, and WebShop. It also achieves the highest recovery rate after local action replacement and higher task success and coverage than successful-only imitation and decoding-time diversification controls.
Tags
Links
- Source: https://arxiv.org/abs/2609.10052v1
- Canonical: https://arxiv.org/abs/2609.10052v1
Trouble viewing inline? Open PDF directly →
Full Text
74,309 characters extracted from source content.
Expand or collapse full text
Direct Diversity Optimization for Diverse Successful Trajectories in Preference Post-Training Junwon Ko Dong-Jae Lee Minchan Kwon Sunghyun Baek Junmo Kim Affiliation: School of Electrical Engineering, KAIST Affiliation: Daejeon, Republic of Korea Email: kojunewon,jhtwosun,kmc0207,baeksh@kaist.ac.kr Email: junmo.kim@kaist.ac.kr Abstract LLM agents for sequential decision tasks are often post-trained with trajectory-level outcome labels, but such labels provide little supervision for preserving multiple successful branches from the same decision state. We study this problem as successful strategy coverage: how broadly a model realizes distinct successful strategies under a fixed rollout budget. We present Direct Diversity Optimization (DDO), an offline post-training method that combines Divergence-Tree Collection (DTC) with the Reference-Relative Target-Odds Objective (RTO).11 1 Code is available at https://github.com/koguma00/direct_diverse_optimization. DTC constructs state-aligned branch sets rooted at shared decision states, and RTO trains the model to match reference-relative targets over successful alternatives. DDO achieves the strongest task success and successful strategy coverage among the compared post-training methods across BabyAI, BabaIsAI, and WebShop. It also achieves the highest recovery rate after local action replacement and higher task success and coverage than successful-only imitation and decoding-time diversification controls. 1 Introduction Recent work on Large Language Models (LLMs) commonly adapts pretrained models to downstream tasks through post-training (Ouyang et al., 2022; Touvron et al., 2023; Grattafiori et al., 2024; Yang et al., 2025). The same approach is now applied to LLM agents in sequential decision tasks such as web navigation, embodied control, and tool use (Zeng et al., 2024; Song et al., 2024; Wei et al., 2025). In these tasks, dense per-step supervision is rarely available, and post-training typically relies on trajectory-level success-or-failure feedback. Under such coarse supervision, standard post-training tends to narrow the model’s output distribution (Kirk et al., 2024; Slocum et al., 2025; Lanchantin et al., 2025). In sequential decision tasks, this narrowing can cause the policy to collapse onto a single successful trajectory even when several viable alternatives remain. Figure 1: A motivating online-shopping example: Model A, a standard post-trained model, completes the purchase through a single trajectory, while Model B retains multiple successful trajectories that can be used as fallbacks when conditions change. Figure 1 illustrates this collapse in an online-shopping task where an LLM agent must buy a laptop charger under a price constraint. A standard post-trained model (Model A) commits to a single brand-page trajectory, whereas a model trained to retain multiple successful trajectories (Model B) can succeed through distinct search, filtering, and comparison strategies. Both models succeed under normal conditions. When the brand-page trajectory is blocked, Model A has no fallback, whereas Model B retains an alternative path to success. The contrast between Models A and B highlights a limitation of standard preference post-training: it lacks state-aligned supervision over alternative branches. Trajectory-level success labels identify a successful trajectory, but do not show the outcomes of alternative actions at the same decision state. Consequently, DPO-style training (Rafailov et al., 2023) with success–failure trajectory preferences distinguishes successful from failed trajectories but provides no explicit target for allocating probability among multiple successful branches from the same state. The learned policy is therefore prone to placing most of its probability mass on one successful branch, leaving little for the remaining alternatives. To address this limitation, we present Direct Diversity Optimization (DDO), an offline post-training method that couples a data-construction procedure with a reference-relative optimization objective. Given a successful source trajectory, Divergence-Tree Collection (DTC) restores an intermediate decision state and constructs an outcome-labeled branch set containing the source and alternative branches. Reference-Relative Target-Odds (RTO) replaces the single-winner preference target with a reference-relative target distribution over successful branches and fits the model to match its induced pairwise odds. Together, DTC and RTO convert trajectory-level outcomes into preference supervision over multiple successful alternatives from the same decision state. We evaluate DDO on BabyAI, BabaIsAI, and WebShop, which cover navigation, object interaction, multi-stage instruction following, rule manipulation, and web interaction. Across the three benchmarks, DDO achieves the strongest joint performance on task success and successful strategy coverage among the compared post-training methods. These gains also translate into more robust recovery after local action replacement. DDO also achieves higher task success and coverage than successful-only imitation and decoding-time diversification controls. Component analyses show complementary roles for DTC and RTO, whose combination yields the largest joint gains. 2 Related Work 2.1 Diversity- and tie-aware post-training. Preference- or reward-based post-training is now a standard mechanism for adapting LLMs with feedback (Stiennon et al., 2020; Ouyang et al., 2022; Bai et al., 2022; Rafailov et al., 2023). However, it has also been shown to narrow the output distribution of trained models: Kirk et al. (2024) and Padmakumar and He (2024) report that RLHF and DPO reduce output diversity relative to the base model, and Slocum et al. (2025) show that KL-regularized preference learning can amplify dominant preferences. To mitigate this collapse, recent approaches preserve multiple acceptable responses in the preference signal. DivPO encourages diverse preferred responses through diversity-aware preference updates (Lanchantin et al., 2025), while tie-aware DPO variants allow similarly preferred responses to be modeled as ties rather than forcing an arbitrary winner–loser direction (Chen et al., 2025). These approaches reduce the pressure to collapse acceptable outputs into a single mode, but they operate at the prompt-response or pairwise-comparison level. Related goals have long been studied in reinforcement learning: maximum-entropy RL encourages stochastic high-return behavior, while skill-discovery and quality-diversity methods learn distinct skills or policy repertoires (Haarnoja et al., 2018; Eysenbach et al., 2019; Pugh et al., 2016; Pierrot et al., 2022). DDO targets offline preference post-training for sequential LLM agents, using state-aligned outcome-labeled branches and reference-relative targets to retain multiple observed successful branches. 2.2 Post-training LLMs for sequential tasks. Preference-based post-training has been extended to sequential decision tasks using branching rollouts or step-level feedback. Chain of Preference Optimization (Zhang et al., 2024) constructs preference data from alternative reasoning trajectories, while ETO (Song et al., 2024) contrasts successful and failed exploration trajectories at the trajectory level. Other methods provide per-step signal: IPR (Xiong et al., 2024) builds step-level contrastive pairs along expert trajectories, and AgentPRM (Xi et al., 2026) trains a process reward model for step-wise progress. Across these methods, the supervision signal primarily distinguishes better from worse trajectories or steps. When multiple branches from the same state are successful, they are not explicitly modeled as a set of valid alternatives; they are typically paired against failures, folded into independent comparisons, or scored by a progress signal. DDO uses the labeled branch set at a decision state as the supervision unit. It preserves multiple successful branches as valid positives while distinguishing them from failed branches, without forcing an arbitrary ranking among the successful ones. This couples diversity preservation directly with task success. Figure 2: DDO pipeline. DTC constructs state-aligned branch sets, and RTO defines a reference-relative target distribution whose induced pairwise targets are optimized with a soft logistic objective. 3 Method We introduce Direct Diversity Optimization (DDO), an offline post-training method for retaining multiple successful strategies in LLM agents trained from trajectory-level outcome labels. DDO couples Divergence-Tree Collection (DTC; Section 3.2), which constructs state-aligned branch sets with rollout outcome labels, with the Reference-Relative Target-Odds Objective (RTO; Section 3.3), which fits the model to reference-relative targets over successful branches. This coupling transforms trajectory-level outcome supervision into same-state branch comparisons and a target distribution over successful alternatives. Figure 2 summarizes the pipeline. 3.1 Problem Setting and Notations We address post-training of LLM agents on sequential decision tasks with textual observations and executable actions. At each step t, the agent observes a textual state xtx_t that includes the current observation and the recent interaction history, and produces an output yty_t; the environment executes the action parsed from yty_t to produce the next state. A trajectory τ=(x0,y0,…,yT−1,xT)τ=(x_0,y_0,…,y_T-1,x_T) is judged only at termination, receiving a binary result R(τ)∈0,1R(τ)∈\0,1\ provided by the environment. This trajectory-level signal is the sole supervision; no per-step rewards or learned reward model are assumed. We write πθ(y∣x) _θ(y x) for the trainable model and πref(y∣x) _ref(y x) for the frozen reference model. 3.2 Divergence-Tree Collection DTC takes a successful trajectory and produces, at selected decision steps, branch sets rooted at a shared state with rollout outcome labels. For a source trajectory τ, the candidate divergence steps are the interior decision points 1,…,|τ|−11,…,|τ|-1. Under a fixed collection budget, DTC selects divergence steps to cover different depths along the source, so that branch roots are spread along the trajectory rather than concentrated near its initial prefix. At each selected divergence step, DTC restores the corresponding decision state as the shared state x (Algorithm 1). It keeps the source output as one branch and queries an expert model for alternative outputs whose parsed actions differ from the source action. Each retained output is executed to termination by the expert model and receives a binary success label. If the state cannot be restored or no distinct valid alternative is obtained, DTC skips that state. Algorithm 1 Divergence-Tree Collection (DTC). Implementation details of Select, Restore, Alt, and Reject are provided in Appendix B.2. 1: Input: successful sources +T^+, expert model E, budgets Kd,KaK_d,K_a 2: Output: DTC records DTCD_DTC 3: DTC←∅D_DTC← 4: for τ∈+τ ^+ do 5: D←Select(1,…,|τ|−1,Kd)D← Select(\1,…,|τ|-1\,K_d) 6: for d∈Dd∈ D do 7: x←Restore(τ,d)x← Restore(τ,d) 8: if x=⊥x= then 9: continue 10: B←jsrcB←\j_src\, yjsrc←ydτy_j_src← y_d^τ, rjsrc←1r_j_src← 1 11: ←Alt(E,x,ydτ,Ka)Y← Alt(E,x,y_d^τ,K_a) 12: for y∈y do 13: if Reject(y,B) Reject(y,B) then 14: continue 15: r←R(Rollout(E,x,y))r← R( Rollout(E,x,y)) 16: add new j with (yj,rj)←(y,r)(y_j,r_j)←(y,r) to B 17: if B≠jsrcB≠\j_src\ then 18: add Record(x,B) Record(x,B) to DTCD_DTC 19: return DTCD_DTC Each collected branch set is represented as one DTC record: C=(x,B,(yj,rj)j∈B).C=(x,B,\(y_j,r_j)\_j∈ B). (1) Here, x is the shared decision state, B is a finite set of branch indices, yjy_j is the LLM output of branch j, and rj∈0,1r_j∈\0,1\ is the binary result obtained after executing that branch. For each record we partition B into successful branches S+=j:rj=1S^+=\j:r_j=1\ and non-successful branches S0=j:rj=0S^0=\j:r_j=0\. RTO uses S+S^+ for target-odds estimation and S0S^0 for boundary comparisons between successful and non-successful observed branches. 3.3 Reference-Relative Target-Odds Objective Pairwise preference objectives that contrast a single winner against a single loser with hard binary labels risk collapsing πθ _θ onto a single successful strategy. The Reference-Relative Target-Odds (RTO) objective addresses this by aligning the log-odds πθ _θ assigns to pairs of successful branches with the log-odds prescribed by a fixed distribution q over S+S^+. For each record C=(x,B,(yj,rj)j∈B)C=(x,B,\(y_j,r_j)\_j∈ B), RTO constructs q from πref _ref and matches the two log-odds in reference-relative form, aligning πθ _θ’s shift over πref _ref with q’s shift over qrefq_ref. Two pair types are handled asymmetrically: success–success pairs use q to set the desired log-odds within S+S^+, and success–failure pairs across S+S^+ and S0S^0 use task success labels to order successful branches above failed ones. Branch distribution. On S+S^+, define the renormalized reference distribution qrefq_ref as qref(j)=πref(yj∣x)∑h∈S+πref(yh∣x).q_ref(j)= _ref(y_j x) _h∈ S^+ _ref(y_h x). (2) The distribution q is defined as q(j)=qref(j)α∑h∈S+qref(h)α,q(j)= q_ref(j)^α _h∈ S^+q_ref(h)^α, (3) where α∈[0,1]α∈[0,1] controls how strongly q preserves πref _ref’s relative weighting among successful branches: smaller α flattens these differences, larger α keeps them. At α=0α=0, q is uniform on S+S^+, providing maximal within-success diversity pressure; at α=1α=1, q=qrefq=q_ref and the success–success targets reduce to preserving πref _ref’s relative odds among successful branches. Both q and qrefq_ref are computed once from πref _ref and held fixed during training. We use α=0.5α=0.5 unless otherwise stated. Distribution margin. For an ordered pair u,v∈S+u,v∈ S^+ of successful branches, the target–reference log-odds margin is given by m⋆=logq(u)q(v)−logqref(u)qref(v).m = q(u)q(v)- q_ref(u)q_ref(v). (4) The margin m⋆m is therefore the reference-relative log-odds shift needed for πθ _θ to match q at (u,v)(u,v). The corresponding target is p⋆=σ(βm⋆)p =σ(β\,m ) with logistic scale β>0β>0; for a success–failure pair with u∈S+u∈ S^+ and v∈S0v∈ S^0, q is undefined on v and we set p⋆=1p =1 directly, so task success labels enter the objective only through these pairs. Model margin. For πθ _θ, the analogous margin is defined in the same form so that it can be directly compared to m⋆m : mθ=logπθ(yu∣x)πθ(yv∣x)−logπref(yu∣x)πref(yv∣x).m_θ= _θ(y_u x) _θ(y_v x)- _ref(y_u x) _ref(y_v x). (5) Aligning mθm_θ with m⋆m is equivalent to matching πθ _θ’s log-odds at (u,v)(u,v) to q’s, which is what the loss enforces. Objective function. We train πθ _θ by minimizing ℒDDO(θ)=[DKL(P⋆∥Pθ)],L_DDO(θ)=E [D_KL (P \,\|\,P_θ ) ], (6) where P⋆=Bern(p⋆)P =Bern(p ) and Pθ=Bern(pθ)P_θ=Bern(p_θ) are Bernoulli distributions with parameters p⋆p and pθ=σ(βmθ)p_θ=σ(β\,m_θ). The expectation is estimated over materialized pairs: each unordered success–success pair is included once, success–failure pairs are oriented toward the successful branch, and state-normalized weights prevent larger branch sets from dominating. Up to a θ-independent constant, this is equivalent to soft binary cross-entropy with target p⋆p and prediction pθp_θ. The success–success terms match the target odds q(u)/q(v)q(u)/q(v) between successful branches, while success–failure terms set p⋆=1p =1 and reduce to the DPO-style loss −logσ(βmθ)- σ(β m_θ), with u∈S+u∈ S^+ and v∈S0v∈ S^0. 4 Experimental Setup 4.1 Benchmarks We evaluate DDO on three benchmarks that span complementary forms of sequential text-agent behavior. BabyAI (Chevalier-Boisvert et al., 2019) probes controlled navigation, object interaction, and multi-stage instruction following. BabaIsAI (Cloos et al., 2024) targets rule manipulation, where the agent must change or exploit environment dynamics. For BabyAI and BabaIsAI, we use the BALROG text-action implementations (Paglieri et al., 2025), which provide standardized textual observations and executable action interfaces. WebShop (Yao et al., 2022) examines web-style shopping behavior involving search, product inspection, option selection, and purchase decisions. Details of each benchmark are provided in Appendix C.1. 4.2 Baselines and Comparison Protocol We compare DDO against DPO and diversity-aware post-training baselines. DPO is the standard Direct Preference Optimization baseline (Rafailov et al., 2023). DivFreq and DivProb adapt DivPO to state-aligned branch data by treating each shared state as a prompt, its executable branch outputs as candidate responses, and terminal rollout outcomes as quality labels (Lanchantin et al., 2025). They retain DivPO’s frequency- and probability-based pair-selection criteria, respectively. TieDPO-RK and TieDPO-Dav are tie-aware DPO baselines based on the Rao-Kupper and Davidson variants of Chen et al. (2025). We use Qwen3-1.7B as the target model and Qwen3.5-122B-A10B-FP8 as the expert model for DTC collection (Yang et al., 2025; Qwen Team, 2026). Base denotes the unadapted target model before task SFT or preference optimization. Reference denotes the frozen task-SFT model, fine-tuned from the base on the successful trajectories that also serve as DTC sources. It is used both as the initialization for all post-training methods and as the reference model for reference-relative objectives. For fair comparison, all post-training methods within a benchmark use the same DTC-collected branch sets, so the comparison isolates objective-side differences while holding the data side fixed. The DTC ablation in Section 5.1 separately varies the data side by replacing this resource with comparisons formed without DTC, allowing us to test the contribution of the collection procedure itself. Training and evaluation otherwise follow the same task budgets and decoding protocols within each benchmark. Full experimental details are provided in Appendix C. 4.3 Metrics We evaluate each model along two axes: task success and successful strategy coverage under a fixed rollout budget. A rollout is valid if it terminates without system failures or missing model outputs. Success rate is the fraction of valid rollouts that solve the task. BabyAI and BabaIsAI count a rollout as successful when the benchmark progression reaches 1.01.0. WebShop returns a graded purchase score rather than a binary success label; we treat scores of at least 0.90.9 as successful, using the same fixed threshold across all methods. To measure successful strategy coverage, we group successful rollouts into trajectory classes using a per-benchmark equivalence relation that abstracts surface variation in the action sequence (see Appendix C.1 for the per-benchmark rules). Two successful rollouts belong to the same class when they are equivalent under this relation. Within each benchmark, we apply the same equivalence relation to the rollouts from all evaluated methods. We use effective strategy diversity (ESD) and entropy effective strategy diversity (H-ESD) to summarize the resulting classes. Both metrics are success-restricted and budget-normalized: they are computed from successful trajectory classes but normalized by the total number of valid rollouts. For each evaluation item i, let KiK_i be the number of valid rollouts and UiU_i the number of unique successful trajectory classes. Let HiH_i be the entropy of the empirical distribution over these classes: ESD(i)=UiKi,H-ESD(i)=2HiKi.ESD(i)= U_iK_i, -ESD(i)= 2^H_iK_i. ESD counts how many distinct successful trajectory classes are observed under the rollout budget, while H-ESD is smaller when successful rollouts concentrate on a few repeated trajectory classes. Normalizing by KiK_i, rather than by the number of successful rollouts, makes the metric reflect finite-budget successful strategy coverage: a model receives a high score only when it both solves the task and produces distinct successful trajectory classes. Reported scores are uniform means of the item-level ESD and H-ESD over the evaluation items in each task. Both metrics are zero when no successful rollout is observed. 5 Results We first evaluate DTC as a data-construction procedure by measuring collection-time successful strategy coverage and downstream performance with and without DTC. We then fix DTC and compare post-training objectives across the three benchmarks, isolating RTO under the same state-aligned branch supervision. 5.1 DTC Improves Branch Supervision Figure 3: Effect of DTC on BabyAI and BabaIsAI. Each curve reports the task-averaged number of unique successful trajectories at a matched per-task request budget, under benchmark-specific trajectory normalization. To isolate the data side of DDO, we run DPO and DDO with and without DTC. DTC converts trajectory-level outcome labels into same-state branch supervision. For the collection-efficiency comparison, Without DTC independently samples full trajectories with the same expert under the matched request budget. For downstream training, settings without DTC form same-task comparisons from separately sampled rollouts matched to the training comparison exposure. Figure 3 shows that DTC discovers more unique successful trajectories under the same request budget on both BabyAI and BabaIsAI, thereby increasing collection-time successful strategy coverage before post-training begins. Figure 4: Distribution of normalized pair-formation positions along source trajectories with and without DTC. Positions are normalized by trajectory length. DTC pairs are distributed across the trajectory, with 56.4%56.4\% formed at or beyond the midpoint. Pairs without DTC are concentrated near the initial prefix: 79.3%79.3\% occur within the first 20%20\% of the source trajectory. This structural difference matters for sequential strategies, because early, middle, and late decisions can play different roles in reaching success. DTC therefore provides branch-level supervision across a broader range of decision depths. Method DTC BabyAI BabaIsAI Succ. ↑ H-ESD ↑ ESD ↑ Succ. ↑ H-ESD ↑ ESD ↑ DPO × 0.76 0.38 0.44 0.78 0.26 0.30 DPO ✓ 0.84 0.36 0.43 0.94 0.31 0.38 DDO × 0.87 0.39 0.46 0.83 0.28 0.35 DDO ✓ 0.92 0.45 0.52 0.99 0.38 0.47 Table 1: Effect of DTC on DPO and DDO. Method Success rate ↑ H-ESD ↑ ESD ↑ BabyAI BabaIsAI WebShop Avg. BabyAI BabaIsAI WebShop Avg. BabyAI BabaIsAI WebShop Avg. Base 0.32 0.21 0.18 0.24 0.28 0.08 0.02 0.13 0.29 0.09 0.02 0.13 Reference 0.82 0.90 0.26 0.66 0.41 0.37 0.15 0.31 0.49 0.43 0.17 0.36 DPO 0.84 0.94 0.26 0.68 0.36 0.31 0.11 0.26 0.43 0.38 0.12 0.31 DivFreq 0.85 0.91 0.26 0.67 0.37 0.30 0.12 0.26 0.44 0.39 0.13 0.32 DivProb 0.89 0.93 0.26 0.69 0.38 0.36 0.11 0.28 0.44 0.43 0.12 0.33 TieDPO-RK 0.88 0.80 0.28 0.65 0.35 0.28 0.16 0.26 0.40 0.35 0.17 0.31 TieDPO-Dav 0.86 0.77 0.16 0.60 0.38 0.30 0.13 0.27 0.43 0.38 0.13 0.31 DDO 0.92 0.99 0.36 0.76 0.45 0.38 0.21 0.35 0.52 0.47 0.22 0.40 Table 2: Main results across BabyAI, BabaIsAI, and WebShop. BabyAI and BabaIsAI report task averages, WebShop reports the shopping evaluation summary, and Avg. is the unweighted mean across benchmarks. These collection differences translate into post-training results. Table 1 compares DPO and DDO with and without DTC. For DPO, adding DTC raises task success by 88 percentage points (p) on BabyAI and 1616 p on BabaIsAI, although its effect on coverage is mixed on BabyAI. For DDO, adding DTC raises task success by 55 p on BabyAI and 1616 p on BabaIsAI. Adding DTC also improves DDO’s coverage metrics: on BabyAI, H-ESD and ESD each increase by 0.060.06; on BabaIsAI, they increase by 0.100.10 and 0.120.12, respectively. Averaged across the two benchmarks, adding DTC to DDO improves task success by 10.510.5 p, H-ESD by 0.080.08, and ESD by 0.090.09. DTC broadens the collected set of successful trajectories and improves downstream post-training. Under matched request budgets, DTC discovers more unique successful trajectories and provides branch supervision at decision states spanning early, middle, and late portions of the source trajectories. The factorization in Table 1 shows complementary contributions from DTC and RTO: DTC expands branch supervision and predominantly raises success, whereas RTO improves both success and coverage under either collection condition. Combining DTC and RTO yields the strongest joint success and coverage result on both BabyAI and BabaIsAI. We next fix the DTC resource and compare post-training objectives under the same state-aligned branch supervision. 5.2 RTO Improves the Success–Coverage Frontier With DTC fixed, we compare post-training objectives on the same state-aligned branch sets. Table 2 reports benchmark averages; full tables appear in Appendix A.6. The three benchmarks probe different aspects of successful strategy coverage—diverse successful action trajectories for the same instruction (BabyAI), diverse exploitable rule configurations (BabaIsAI), and diverse shopping behaviors ending in valid purchases (WebShop). Joint success and coverage gains. DDO achieves the highest benchmark-average success (0.760.76), H-ESD (0.350.35), and ESD (0.400.40). Relative to the DPO row, these values correspond to gains of 88 p, 0.090.09, and 0.090.09, respectively. The largest relative gain appears on WebShop, where H-ESD nearly doubles from 0.110.11 to 0.210.21 and ESD nearly doubles from 0.120.12 to 0.220.22, alongside a 1010 p success gain from 0.260.26 to 0.360.36. Because WebShop trajectory classes incorporate purchase realization, the gain reflects broader successful strategy coverage under the composite class definition. Coverage relative to task-SFT initialization. DDO raises task success by 1010 p and both coverage metrics by 0.040.04 over the shared Reference initialization. The remaining post-training methods stay below Reference on both coverage metrics; DPO, for example, raises average success from 0.660.66 to 0.680.68 while H-ESD and ESD each fall by 0.050.05. Other post-training variants. Across the compared methods, DDO leads every benchmark-level aggregate and all three overall averages. DivFreq and DivProb, which filter pairs by frequency- or probability-based diversity, change success and coverage by at most ±0.02± 0.02 on average. TieDPO-RK and TieDPO-Dav reach lower average success than DPO (−3-3 and −8-8 p), with the largest drops on BabaIsAI (−0.14-0.14 and −0.17-0.17 success) and, for TieDPO-Dav, WebShop (−0.10-0.10); their coverage matches DPO on average. 6 Analysis 6.1 Strategic Recovery We use strategic recovery to test whether broader successful strategy coverage provides alternative routes after a local disruption. For each successful source trajectory, we first sample one interior decision point. We then replace the source action at that point with a different valid task action and roll out the same model from the edited prefix. Because every source trajectory solves the task before the edit, the score measures recovery from the local action replacement rather than ordinary task success. Appendix C details the probe construction. DDO achieves the highest recovery rate, 75.2%75.2\%, compared with 70.1%70.1\% for DivFreq and 69.7%69.7\% for DPO. After the initial trajectory is disrupted, DDO more often finds another viable trajectory to success, showing that its broader successful strategy coverage is accompanied by higher recovery. Figure 5: Strategic recovery on BabyAI after local action replacement in successful source trajectories. 6.2 Imitation Control To separate successful-branch exposure from same-state outcome comparisons and reference-relative targets, we train successful-only SFT models on the same successful DTC branches used by DDO. We compare DDO with two imitation settings matched by optimizer steps and nominal epochs, respectively. At matched optimizer steps, successful-only imitation reaches H-ESD 0.390.39 and ESD 0.440.44, compared with DDO’s 0.420.42 and 0.500.50, while DDO retains a 3232 p advantage in task success. At 7.30×7.30× exposure, imitation reaches H-ESD 0.410.41 and ESD 0.470.47, while DDO retains a 2424 p advantage in task success. Successful-only SFT broadens coverage, while DDO retains a 2424–3232 p advantage in task success under both matching conditions. 6.3 Decoding Control We also test whether inference-time diversification can recover successful strategy coverage without post-training changes. We compare DDO against DPO under a range of sampling temperatures. Method Exposure Success ↑ H-ESD ↑ ESD ↑ DDO, 5 ep. 1.00×1.00× 0.96 0.42 0.50 Imit. matched 1.00×1.00× 0.64 0.39 0.44 Imit. 5 ep. 7.30×7.30× 0.72 0.41 0.47 Table 3: Successful-only imitation control averaged over BabyAI and BabaIsAI. Method T Success ↑ H-ESD ↑ ESD ↑ DDO 0.6 0.76 0.35 0.40 DPO + temp sweep 0.0 0.60 0.22 0.25 DPO + temp sweep 0.1 0.62 0.20 0.24 DPO + temp sweep 0.6 0.68 0.26 0.31 DPO + temp sweep 1.0 0.66 0.29 0.35 DPO + temp sweep 1.5 0.67 0.33 0.39 Table 4: DPO temperature sweep averaged across BabyAI, BabaIsAI, and WebShop; DDO at T=0.6T=0.6 is shown for comparison. Simply increasing DPO’s sampling temperature raises strategy coverage, but the gain comes with lower task success. At T=1.5T=1.5, DPO reaches an ESD of 0.390.39, close to DDO’s 0.400.40, while DDO retains a 99 p advantage in task success. Increasing temperature therefore recovers coverage by sacrificing success, whereas DDO improves the joint success and coverage result through post-training. 7 Conclusion We present Direct Diversity Optimization (DDO), an offline post-training method for preserving multiple successful strategies in LLM agents trained from trajectory-level outcome labels. DDO consists of two components: Divergence-Tree Collection (DTC) and the Reference-Relative Target-Odds Objective (RTO). DTC builds state-aligned branch sets with per-branch outcome labels, and RTO trains the model toward reference-relative targets over successful branches. Across all benchmarks, DDO achieves the strongest joint performance in task success and successful strategy coverage among the compared post-training methods. DDO also achieves the highest recovery rate in the local action replacement evaluation. Compared with successful-only imitation and decoding-time diversification, DDO achieves broader coverage together with higher task success by training on same-state outcome comparisons with reference-relative targets. These results establish DDO as a training-time method that improves task success while broadening successful strategy coverage in sequential decision tasks. Limitations For controlled benchmarking and method comparison, our experiments are confined to environments that support state reconstruction and the execution of alternative branches from a shared decision state under a common success predicate. This setting makes the outcomes of same-state alternative branches directly observable, but it excludes continuous control, partially observable or stochastic dynamics, multimodal observations, and real-world agents whose actions induce non-reversible external side effects. Extending DTC to settings without exact state reconstruction would require approximate state-aligned branch sets. Such sets could be constructed from rollouts generated by learned simulators or world models, or from logged trajectories that contain comparable decision contexts and alternative outcomes. Ethical Considerations License of Existing Assets. The models used in this work, Qwen3-1.7B and Qwen3.5-122B-A10B-FP8, are released under the Apache 2.0 license. The benchmark environments are used under their respective licenses: BabyAI under the BSD 3-Clause License, BabaIsAI under the MIT License, and WebShop under the MIT License with copyright attributed to Princeton Natural Language Processing. All assets and environments were used for academic, non-commercial evaluation purposes and in accordance with the applicable license terms. Acknowledgments This work was supported by the Artificial Intelligence Industrial Convergence Cluster Development Project, funded by the Ministry of Science and ICT (MSIT), Korea, and Gwangju Metropolitan City. References Bai et al. (2022) Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, N. Joseph, S. Kadavath, J. Kernion, T. Conerly, S. El-Showk, N. Elhage, Z. Hatfield-Dodds, D. Hernandez, T. Hume, S. Johnston, S. Kravec, L. Lovitt, N. Nanda, C. Olsson, D. Amodei, T. Brown, J. Clark, S. McCandlish, C. Olah, B. Mann, and J. Kaplan Training a helpful and harmless assistant with reinforcement learning from human feedback. External Links: 2204.05862, Document, Link Cited by: §2.1. Chen et al. (2025) J. Chen, G. Yang, W. Lin, J. Mei, C. Lyu, and B. Byrne On extending direct preference optimization to accommodate ties. In Advances in Neural Information Processing Systems, Vol. 38, Main Conference, p. 53932–53977. External Links: Document, Link Cited by: §2.1, §4.2. Chevalier-Boisvert et al. (2019) M. Chevalier-Boisvert, D. Bahdanau, S. Lahlou, L. Willems, C. Saharia, T. H. Nguyen, and Y. Bengio BabyAI: a platform to study the sample efficiency of grounded language learning. In International Conference on Learning Representations, External Links: Link Cited by: §4.1. Cloos et al. (2024) N. Cloos, M. Jens, M. Naim, Y. Kuo, I. Cases, A. Barbu, and C. J. Cueva Baba is AI: break the rules to beat the benchmark. In ICML 2024 Workshop on LLMs and Cognition, External Links: Link Cited by: §4.1. Eysenbach et al. (2019) B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine Diversity is all you need: learning skills without a reward function. In International Conference on Learning Representations, External Links: Link Cited by: §2.1. Grattafiori et al. (2024) A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sravankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru, B. Roziere, B. Biron, B. Tang, B. Chern, C. Caucheteux, C. Nayak, C. Bi, C. Marra, C. McConnell, C. Keller, C. Touret, C. Wu, C. Wong, C. C. Ferrer, C. Nikolaidis, D. Allonsius, D. Song, D. Pintz, D. Livshits, D. Wyatt, D. Esiobu, D. Choudhary, D. Mahajan, D. Garcia-Olano, D. Perino, D. Hupkes, E. Lakomkin, E. AlBadawy, E. Lobanova, E. Dinan, E. M. Smith, F. Radenovic, F. Guzmán, F. Zhang, G. Synnaeve, G. Lee, G. L. Anderson, G. Thattai, G. Nail, G. Mialon, G. Pang, G. Cucurell, H. Nguyen, H. Korevaar, H. Xu, H. Touvron, I. Zarov, I. A. Ibarra, I. Kloumann, I. Misra, I. Evtimov, J. Zhang, J. Copet, J. Lee, J. Geffert, J. Vranes, J. Park, J. Mahadeokar, J. Shah, J. van der Linde, J. Billock, J. Hong, J. Lee, J. Fu, J. Chi, J. Huang, J. Liu, J. Wang, J. Yu, J. Bitton, J. Spisak, J. Park, J. Rocca, J. Johnstun, J. Saxe, J. Jia, K. V. Alwala, K. Prasad, K. Upasani, K. Plawiak, K. Li, K. Heafield, K. Stone, K. El-Arini, K. Iyer, K. Malik, K. Chiu, K. Bhalla, K. Lakhotia, L. Rantala-Yeary, L. van der Maaten, L. Chen, L. Tan, L. Jenkins, L. Martin, L. Madaan, L. Malo, L. Blecher, L. Landzaat, L. de Oliveira, M. Muzzi, M. Pasupuleti, M. Singh, M. Paluri, M. Kardas, M. Tsimpoukelli, M. Oldham, M. Rita, M. Pavlova, M. Kambadur, M. Lewis, M. Si, M. K. Singh, M. Hassan, N. Goyal, N. Torabi, N. Bashlykov, N. Bogoychev, N. Chatterji, N. Zhang, O. Duchenne, O. Çelebi, P. Alrassy, P. Zhang, P. Li, P. Vasic, P. Weng, P. Bhargava, P. Dubal, P. Krishnan, P. S. Koura, P. Xu, Q. He, Q. Dong, R. Srinivasan, R. Ganapathy, R. Calderer, R. S. Cabral, R. Stojnic, R. Raileanu, R. Maheswari, R. Girdhar, R. Patel, R. Sauvestre, R. Polidoro, R. Sumbaly, R. Taylor, R. Silva, R. Hou, R. Wang, S. Hosseini, S. Chennabasappa, S. Singh, S. Bell, S. S. Kim, S. Edunov, S. Nie, S. Narang, S. Raparthy, S. Shen, S. Wan, S. Bhosale, S. Zhang, S. Vandenhende, S. Batra, S. Whitman, S. Sootla, S. Collot, S. Gururangan, S. Borodinsky, T. Herman, T. Fowler, T. Sheasha, T. Georgiou, T. Scialom, T. Speckbacher, T. Mihaylov, T. Xiao, U. Karn, V. Goswami, V. Gupta, V. Ramanathan, V. Kerkez, V. Gonguet, V. Do, V. Vogeti, V. Albiero, V. Petrovic, W. Chu, W. Xiong, W. Fu, W. Meers, X. Martinet, X. Wang, X. Wang, X. E. Tan, X. Xia, X. Xie, X. Jia, X. Wang, Y. Goldschlag, Y. Gaur, Y. Babaei, Y. Wen, Y. Song, Y. Zhang, Y. Li, Y. Mao, Z. D. Coudert, Z. Yan, Z. Chen, Z. Papakipos, A. Singh, A. Srivastava, A. Jain, A. Kelsey, A. Shajnfeld, A. Gangidi, A. Victoria, A. Goldstand, A. Menon, A. Sharma, A. Boesenberg, A. Baevski, A. Feinstein, A. Kallet, A. Sangani, A. Teo, A. Yunus, A. Lupu, A. Alvarado, A. Caples, A. Gu, A. Ho, A. Poulton, A. Ryan, A. Ramchandani, A. Dong, A. Franco, A. Goyal, A. Saraf, A. Chowdhury, A. Gabriel, A. Bharambe, A. Eisenman, A. Yazdan, B. James, B. Maurer, B. Leonhardi, B. Huang, B. Loyd, B. D. Paola, B. Paranjape, B. Liu, B. Wu, B. Ni, B. Hancock, B. Wasti, B. Spence, B. Stojkovic, B. Gamido, B. Montalvo, C. Parker, C. Burton, C. Mejia, C. Liu, C. Wang, C. Kim, C. Zhou, C. Hu, C. Chu, C. Cai, C. Tindal, C. Feichtenhofer, C. Gao, D. Civin, D. Beaty, D. Kreymer, D. Li, D. Adkins, D. Xu, D. Testuggine, D. David, D. Parikh, D. Liskovich, D. Foss, D. Wang, D. Le, D. Holland, E. Dowling, E. Jamil, E. Montgomery, E. Presani, E. Hahn, E. Wood, E. Le, E. Brinkman, E. Arcaute, E. Dunbar, E. Smothers, F. Sun, F. Kreuk, F. Tian, F. Kokkinos, F. Ozgenel, F. Caggioni, F. Kanayet, F. Seide, G. M. Florez, G. Schwarz, G. Badeer, G. Swee, G. Halpern, G. Herman, G. Sizov, G. Zhang, G. Lakshminarayanan, H. Inan, H. Shojanazeri, H. Zou, H. Wang, H. Zha, H. Habeeb, H. Rudolph, H. Suk, H. Aspegren, H. Goldman, H. Zhan, I. Damlaj, I. Molybog, I. Tufanov, I. Leontiadis, I. Veliche, I. Gat, J. Weissman, J. Geboski, J. Kohli, J. Lam, J. Asher, J. Gaya, J. Marcus, J. Tang, J. Chan, J. Zhen, J. Reizenstein, J. Teboul, J. Zhong, J. Jin, J. Yang, J. Cummings, J. Carvill, J. Shepard, J. McPhie, J. Torres, J. Ginsburg, J. Wang, K. Wu, K. H. U, K. Saxena, K. Khandelwal, K. Zand, K. Matosich, K. Veeraraghavan, K. Michelena, K. Li, K. Jagadeesh, K. Huang, K. Chawla, K. Huang, L. Chen, L. Garg, L. A, L. Silva, L. Bell, L. Zhang, L. Guo, L. Yu, L. Moshkovich, L. Wehrstedt, M. Khabsa, M. Avalani, M. Bhatt, M. Mankus, M. Hasson, M. Lennie, M. Reso, M. Groshev, M. Naumov, M. Lathi, M. Keneally, M. Liu, M. L. Seltzer, M. Valko, M. Restrepo, M. Patel, M. Vyatskov, M. Samvelyan, M. Clark, M. Macey, M. Wang, M. J. Hermoso, M. Metanat, M. Rastegari, M. Bansal, N. Santhanam, N. Parks, N. White, N. Bawa, N. Singhal, N. Egebo, N. Usunier, N. Mehta, N. P. Laptev, N. Dong, N. Cheng, O. Chernoguz, O. Hart, O. Salpekar, O. Kalinli, P. Kent, P. Parekh, P. Saab, P. Balaji, P. Rittner, P. Bontrager, P. Roux, P. Dollar, P. Zvyagina, P. Ratanchandani, P. Yuvraj, Q. Liang, R. Alao, R. Rodriguez, R. Ayub, R. Murthy, R. Nayani, R. Mitra, R. Parthasarathy, R. Li, R. Hogan, R. Battey, R. Wang, R. Howes, R. Rinott, S. Mehta, S. Siby, S. J. Bondu, S. Datta, S. Chugh, S. Hunt, S. Dhillon, S. Sidorov, S. Pan, S. Mahajan, S. Verma, S. Yamamoto, S. Ramaswamy, S. Lindsay, S. Lindsay, S. Feng, S. Lin, S. C. Zha, S. Patil, S. Shankar, S. Zhang, S. Zhang, S. Wang, S. Agarwal, S. Sajuyigbe, S. Chintala, S. Max, S. Chen, S. Kehoe, S. Satterfield, S. Govindaprasad, S. Gupta, S. Deng, S. Cho, S. Virk, S. Subramanian, S. Choudhury, S. Goldman, T. Remez, T. Glaser, T. Best, T. Koehler, T. Robinson, T. Li, T. Zhang, T. Matthews, T. Chou, T. Shaked, V. Vontimitta, V. Ajayi, V. Montanez, V. Mohan, V. S. Kumar, V. Mangla, V. Ionescu, V. Poenaru, V. T. Mihailescu, V. Ivanov, W. Li, W. Wang, W. Jiang, W. Bouaziz, W. Constable, X. Tang, X. Wu, X. Wang, X. Wu, X. Gao, Y. Kleinman, Y. Chen, Y. Hu, Y. Jia, Y. Qi, Y. Li, Y. Zhang, Y. Zhang, Y. Adi, Y. Nam, Y. Wang, Y. Zhao, Y. Hao, Y. Qian, Y. Li, Y. He, Z. Rait, Z. DeVito, Z. Rosnbrick, Z. Wen, Z. Yang, Z. Zhao, and Z. Ma The Llama 3 herd of models. External Links: 2407.21783, Document, Link Cited by: §1. Haarnoja et al. (2018) T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 80, p. 1861–1870. External Links: Link Cited by: §2.1. Hu et al. (2022) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations, External Links: Link Cited by: §C.2. Kirk et al. (2024) R. Kirk, I. Mediratta, C. Nalmpantis, J. Luketina, E. Hambro, E. Grefenstette, and R. Raileanu Understanding the effects of RLHF on LLM generalisation and diversity. In International Conference on Learning Representations, Vol. 2024, p. 20620–20653. External Links: Link Cited by: §1, §2.1. Lanchantin et al. (2025) J. Lanchantin, A. Chen, S. Dhuliawala, P. Yu, J. Weston, S. Sukhbaatar, and I. Kulikov Diverse preference optimization. External Links: 2501.18101, Document, Link Cited by: §1, §2.1, §4.2. Loshchilov and Hutter (2019) I. Loshchilov and F. Hutter Decoupled weight decay regularization. In International Conference on Learning Representations, External Links: Link Cited by: §C.2. Ouyang et al. (2022) L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, Vol. 35, p. 27730–27744. External Links: Document, Link Cited by: §1, §2.1. Padmakumar and He (2024) V. Padmakumar and H. He Does writing with language models reduce content diversity?. In International Conference on Learning Representations, Vol. 2024, p. 642–669. External Links: Link Cited by: §2.1. Paglieri et al. (2025) D. Paglieri, B. Cupiał, S. Coward, U. Piterbarg, M. Wołczyk, A. Khan, E. Pignatelli, Ł. Kuciński, L. Pinto, R. Fergus, J. Foerster, J. Parker-Holder, and T. Rocktäschel BALROG: benchmarking agentic LLM and VLM reasoning on games. In International Conference on Learning Representations, Vol. 2025, p. 96666–96702. External Links: Link Cited by: §4.1. Pierrot et al. (2022) T. Pierrot, V. Macé, F. Chalumeau, A. Flajolet, G. Cideron, K. Beguir, A. Cully, O. Sigaud, and N. Perrin-Gilbert Diversity policy gradient for sample efficient quality-diversity optimization. In Proceedings of the Genetic and Evolutionary Computation Conference, p. 1075–1083. External Links: Document, Link Cited by: §2.1. Pugh et al. (2016) J. K. Pugh, L. B. Soros, and K. O. Stanley Quality diversity: a new frontier for evolutionary computation. Frontiers in Robotics and AI 3, p. 40. External Links: Document, Link Cited by: §2.1. Qwen Team (2026) Qwen Team Qwen3.5: towards native multimodal agents. External Links: Link Cited by: §C.2, §4.2. Rafailov et al. (2023) R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn Direct preference optimization: your language model is secretly a reward model. In Advances in Neural Information Processing Systems, Vol. 36, p. 53728–53741. External Links: Document, Link Cited by: §C.2, §1, §2.1, §4.2. Slocum et al. (2025) S. Slocum, A. Parker-Sartori, and D. Hadfield-Menell Diverse preference learning for capabilities and alignment. In International Conference on Learning Representations, Vol. 2025, p. 24760–24790. External Links: Link Cited by: §1, §2.1. Song et al. (2024) Y. Song, D. Yin, X. Yue, J. Huang, S. Li, and B. Y. Lin Trial and error: exploration-based trajectory optimization of LLM agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand, p. 7584–7600. External Links: Document, Link Cited by: §1, §2.2. Stiennon et al. (2020) N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano Learning to summarize with human feedback. In Advances in Neural Information Processing Systems, Vol. 33, p. 3008–3021. External Links: Link Cited by: §2.1. Touvron et al. (2023) H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom Llama 2: open foundation and fine-tuned chat models. External Links: 2307.09288, Document, Link Cited by: §1. Wei et al. (2025) Z. Wei, W. Yao, Y. Liu, W. Zhang, Q. Lu, L. Qiu, C. Yu, P. Xu, C. Zhang, B. Yin, H. Yun, and L. Li WebAgent-r1: training web agents via end-to-end multi-turn reinforcement learning. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, Suzhou, China, p. 7909–7928. External Links: Document, Link Cited by: §1. Xi et al. (2026) Z. Xi, C. Liao, G. Li, Z. Zhang, W. Chen, B. Wang, S. Jin, Y. Zhou, J. Guan, W. Wu, T. Ji, T. Gui, Q. Zhang, and X. Huang AgentPRM: process reward models for LLM agents via step-wise promise and progress. In Proceedings of the ACM Web Conference 2026, p. 4184–4195. External Links: Document, Link Cited by: §2.2. Xiong et al. (2024) W. Xiong, Y. Song, X. Zhao, W. Wu, X. Wang, K. Wang, C. Li, W. Peng, and S. Li Watch every step! LLM agent learning via iterative step-level process refinement. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Miami, Florida, USA, p. 1556–1572. External Links: Document, Link Cited by: §2.2. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu Qwen3 technical report. External Links: 2505.09388, Document, Link Cited by: §C.2, §1, §4.2. Yao et al. (2022) S. Yao, H. Chen, J. Yang, and K. Narasimhan WebShop: towards scalable real-world web interaction with grounded language agents. In Advances in Neural Information Processing Systems, Vol. 35, p. 20744–20757. External Links: Document, Link Cited by: §4.1. Zeng et al. (2024) A. Zeng, M. Liu, R. Lu, B. Wang, X. Liu, Y. Dong, and J. Tang AgentTuning: enabling generalized agent abilities for LLMs. In Findings of the Association for Computational Linguistics: ACL 2024, L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, p. 3053–3077. External Links: Link, Document Cited by: §1. Zhang et al. (2024) X. Zhang, C. Du, T. Pang, Q. Liu, W. Gao, and M. Lin Chain of preference optimization: improving chain-of-thought reasoning in LLMs. In Advances in Neural Information Processing Systems, Vol. 37, p. 333–356. External Links: Document, Link Cited by: §2.2. Appendix A Additional Results A.1 Model-Scale Sensitivity We vary the expert and target models separately on BabyAI while keeping all other collection, training, and evaluation settings fixed (Table 5). Scale Policy Success ↑ H-ESD ↑ ESD ↑ Expert model scale 122B DDO 0.92 0.45 0.52 35B DDO 0.87 0.41 0.48 Target model scale 1.7B Reference 0.82 0.41 0.49 DDO 0.92 0.45 0.52 4B Reference 0.85 0.37 0.42 DDO 0.91 0.40 0.45 Table 5: Model-scale sensitivity on BabyAI. Each block varies one model while holding the other fixed. Reducing the expert model size from 122B to 35B changes task success from 0.920.92 to 0.870.87, H-ESD from 0.450.45 to 0.410.41, and ESD from 0.520.52 to 0.480.48. Even with the 35B expert model, DDO retains more than 90%90\% of the corresponding 122B value for each metric: 94.6%94.6\% for task success, 91.1%91.1\% for H-ESD, and 92.3%92.3\% for ESD. The number of completed alternative branches also remains similar (3,3293,329 versus 3,4063,406). DDO improves over its corresponding Reference at both target model sizes. For the 1.7B target model, DDO improves task success, H-ESD, and ESD by 0.100.10, 0.040.04, and 0.030.03, respectively. For the 4B target model, the corresponding gains are 0.060.06, 0.030.03, and 0.030.03. A.2 Coverage Growth Beyond final-budget scores, Figure 6 traces the cumulative number of unique successful trajectories found within the first N rollouts on BabyAI, BabaIsAI, and WebShop. DDO stays above the other plotted methods across the sampled budget range, so its coverage gain is visible throughout sampling and at the final evaluation budget. Figure 6: Coverage growth over BabyAI, BabaIsAI, and WebShop. Each curve indicates the cumulative number of unique successful trajectories found within the first N rollouts. A.3 Shared Cross-Task Post-Training The main benchmark uses SFT and preference adapters for each task to provide controlled, matched comparisons. As an additional setting, we test whether DDO remains effective when both the initializer and post-training model are shared across tasks. On BabyAI, a single BabyAI-4 SFT adapter initializes all four tasks, and preference methods are trained on pooled BabyAI-4 preference data. The table reports BabyAI task averages under the same evaluation budgets and decoding settings as the main results. Method Success ↑ H-ESD ↑ ESD ↑ SFT 0.78 0.37 0.41 DPO 0.81 0.38 0.43 DDO 0.92 0.48 0.52 Table 6: Shared cross-task SFT and post-training on BabyAI using one BabyAI-4 SFT initialization and pooled preference data. Table 6 shows that DDO consistently improves both task success and successful strategy coverage in the shared-model setting. DDO achieves 0.920.92 success, 0.480.48 H-ESD, and 0.520.52 ESD, extending its joint success and coverage gains from task-specific adapters to pooled cross-task post-training. A.4 Branch-Set Target Distribution In Eq. (3), we interpret the RTO target as a log-odds interpolation between two endpoints: a uniform distribution on S+S^+ (α=0α=0, encoding within-success coverage) and the reference distribution restricted to S+S^+ (α=1α=1, preserving the relative ordering inherited from πref _ref). The intermediate setting α=0.5α=0.5 retains both signals. Table 7 compares these three regimes on BabyAI. α Success ↑ H-ESD ↑ ESD ↑ 1.0 0.90 0.39 0.42 0.5 0.92 0.45 0.52 0.0 0.87 0.42 0.48 Table 7: Effect of the target-distribution coefficient α on BabyAI. The intermediate target (α=0.5α=0.5) achieves the highest success (0.920.92), H-ESD (0.450.45), and ESD (0.520.52). At the endpoints, α=1α=1 preserves the reference-relative weighting over successful branches, while α=0α=0 uses a uniform target. The intermediate setting combines both target components and yields the strongest joint success and coverage result. A.5 Margin Sharpness Sensitivity The inverse-temperature parameter β controls the sharpness of the pairwise margin loss, determining how strongly deviations from the target pair odds are penalized. Figure 7: Effect of the margin sharpness β on DDO, averaged over BabyAI and BabaIsAI. Axes report success rate and H-ESD. Figure 7 reports the DDO β sweep over 0.1,0.5,1.0\0.1,0.5,1.0\. Across this range, success changes only mildly, while H-ESD remains nearly unchanged after rounding. DDO therefore varies little with margin sharpness across the tested values. A.6 Full Main Benchmark Tables We provide the full benchmark results in Tables 8, 9 and 10. Bold marks the strongest preference-trained result in each column, including ties. BabyAI. DDO leads on average success rate, H-ESD, and ESD. The task columns show the same pattern: success and H-ESD increase for Goto, Pick, Open, and Comp, and ESD increases on three tasks and reaches 0.570.57 on Comp. The Open and Comp columns make this pattern visible in tasks involving object interaction and multi-step composition; DDO improves both success and coverage there, so broader successful strategy coverage accompanies higher task success. BabaIsAI. DDO is again the strongest method on average success rate, H-ESD, and ESD, with gains across Basic, Room, Stop, and Flex. The Stop column is particularly informative: DDO raises success to 1.001.00 while also improving both coverage metrics, indicating that the gains extend to rule manipulation tasks where successful behavior depends on changing the rule structure. DivProb and the tie-aware baselines are strongest or tied in a few individual coverage columns. DDO leads the benchmark averages while improving success and coverage together. WebShop. DDO is the strongest method across all three reported metrics. Because WebShop counts unique successful trajectories using both trajectory structure and purchase realization, the gains reflect broader successful strategy coverage under the composite trajectory-class definition. Method Success rate ↑ H-ESD ↑ ESD ↑ Goto. Pick. Open. Comp. Avg. Goto. Pick. Open. Comp. Avg. Goto. Pick. Open. Comp. Avg. Base 0.76 0.22 0.04 0.26 0.32 0.50 0.28 0.02 0.30 0.28 0.55 0.28 0.02 0.30 0.29 Reference 0.90 0.94 0.69 0.74 0.82 0.14 0.36 0.67 0.46 0.41 0.27 0.47 0.67 0.57 0.49 DPO 0.90 0.92 0.72 0.82 0.84 0.09 0.15 0.76 0.45 0.36 0.15 0.23 0.77 0.57 0.43 DivFreq 0.94 0.96 0.70 0.80 0.85 0.13 0.19 0.75 0.40 0.37 0.18 0.30 0.75 0.52 0.44 DivProb 0.94 0.98 0.78 0.86 0.89 0.13 0.16 0.79 0.45 0.38 0.18 0.22 0.80 0.55 0.44 TieDPO-RK 1.00 1.00 0.74 0.76 0.88 0.11 0.17 0.70 0.43 0.35 0.17 0.27 0.70 0.47 0.40 TieDPO-Dav 0.94 1.00 0.64 0.86 0.86 0.10 0.26 0.72 0.45 0.38 0.18 0.33 0.72 0.50 0.43 DDO 0.94 1.00 0.80 0.92 0.92 0.21 0.24 0.84 0.51 0.45 0.30 0.35 0.85 0.57 0.52 Table 8: Main results on BabyAI. Avg. is the unweighted mean across the four tasks. Method Success rate ↑ H-ESD ↑ ESD ↑ Basic. Room. Stop. Flex. Avg. Basic. Room. Stop. Flex. Avg. Basic. Room. Stop. Flex. Avg. Base 0.22 0.26 0.08 0.26 0.21 0.02 0.05 0.06 0.19 0.08 0.02 0.05 0.07 0.22 0.09 Reference 0.94 0.97 0.86 0.84 0.90 0.20 0.11 0.59 0.59 0.37 0.29 0.17 0.62 0.64 0.43 DPO 0.96 0.92 0.92 0.96 0.94 0.10 0.06 0.58 0.50 0.31 0.19 0.10 0.65 0.59 0.38 DivFreq 0.94 0.92 0.78 0.98 0.91 0.13 0.07 0.58 0.42 0.30 0.25 0.12 0.71 0.47 0.39 DivProb 0.96 0.96 0.84 0.96 0.93 0.13 0.06 0.68 0.56 0.36 0.23 0.08 0.78 0.63 0.43 TieDPO-RK 0.94 0.90 0.60 0.76 0.80 0.14 0.07 0.45 0.44 0.28 0.25 0.10 0.53 0.52 0.35 TieDPO-Dav 0.90 0.94 0.50 0.74 0.77 0.19 0.07 0.42 0.54 0.30 0.28 0.10 0.50 0.62 0.38 DDO 0.98 0.98 1.00 0.98 0.99 0.18 0.07 0.75 0.52 0.38 0.30 0.13 0.81 0.63 0.47 Table 9: Main results on BabaIsAI. Avg. is the unweighted mean across the four tasks. Method Success rate ↑ H-ESD ↑ ESD ↑ Base 0.18 0.02 0.02 Reference 0.26 0.15 0.17 DPO 0.26 0.11 0.12 DivFreq 0.26 0.12 0.13 DivProb 0.26 0.11 0.12 TieDPO-RK 0.28 0.16 0.17 TieDPO-Dav 0.16 0.13 0.13 DDO 0.36 0.21 0.22 Table 10: Main results on WebShop. Appendix B Additional Method Details B.1 Action Parsing and Branch Output Probabilities At each environment step, the LLM emits a model output yty_t, and a benchmark-specific parser extracts the executable action for the environment. Some prompts structure yty_t as reasoning text followed by an action field; DDO assigns sequence probability to the complete model output and uses the parsed action for execution. For a DTC record C=(x,B,(yj,rj)j∈B)C=(x,B,\(y_j,r_j)\_j∈ B), πθ(yj∣x) _θ(y_j x) is the sequence probability of the model output under standard left-to-right token factorization. The rollout determines the branch outcome label rjr_j. B.2 DTC Collection Details The helper routines in Algorithm 1 are instantiated as follows. Across benchmarks, we set KdK_d to at most 55 and KaK_a to at most 33; shorter trajectories and branch rejection can yield fewer retained points or alternatives. Select allocates the divergence budget KdK_d over interior decision points 1,…,T−11,…,T-1 so that selected steps cover different trajectory depths. Restore returns the decision state associated with the selected source prefix, or ⊥ when the state cannot be constructed under the benchmark environment. Alt queries the expert model E for at most KaK_a alternative outputs from the shared state x, required to differ from the source output under the benchmark’s executable-action normalization. Reject removes outputs that cannot be executed or that do not provide a distinct executable branch decision, and retained outputs are rolled out by E to obtain outcome labels. Appendix C Experimental Details C.1 Benchmark Details BabyAI provides controlled navigation and object interaction tasks with executable text actions. We use four tasks: goto, pickup, open, and pick-up sequence go-to. They are reported in Table 8 as Goto, Pick, Open, and Comp. For trajectory-class normalization, we collapse modulo-four same-direction turn repetitions and remove alternating turn blocks that cancel out, both of which leave the agent’s pose unchanged. BabaIsAI provides tasks based on rule manipulation with executable text actions. We use four tasks: goto, two-room goto, two-room break-stop goto, and two-room optional break-stop goto. They are reported in Table 9 as Basic, Room, Stop, and Flex. For trajectory-class normalization, we drop non-terminal steps with no observation change and two-step inverse moves that return the agent to a prior state, both of which leave the environment state unchanged. WebShop provides shopping tasks in which an agent searches, inspects products, selects options, and purchases an item for a user instruction. For trajectory-class normalization, the class combines the purchased item, selected options, and normalized trajectory structure, making coverage invariant to surface variation in the action trace. Figure 8 shows how this rule merges query variants along the same route while distinguishing different search and navigation routes to the same purchase. Same class (query variation). Two successful rollouts begin with search[blue coated steel end table] and search[blue-coated steel end table], follow the same normalized trajectory structure, and purchase item B08MF23ZPL with the blue option. Different classes (route variation). Two successful DDO rollouts purchase item B09P572DP9 with the redblack option. Their normalized routes are search → product inspection → refined search → purchase and search → pagination → refined search → purchase. Figure 8: WebShop trajectory-class examples. Example input prompts and LLM outputs for all three benchmarks are shown in Appendix D. C.2 Models and Optimization For the experiments, expert trajectories are collected with Qwen3.5-122B-A10B-FP8 under the shared thought-action format, and all optimized models use Qwen3-1.7B (Yang et al., 2025; Qwen Team, 2026). The FP8 model provides the expert trajectory source. SFT and preference post-training for the target model use bf16 mixed precision. Base denotes the unadapted target model before SFT or preference optimization for each task. For each benchmark task, the Reference model is a shared SFT adapter trained from the base model for 10 epochs. DPO Rafailov et al. (2023), DivFreq, DivProb, TieDPO-RK, TieDPO-Dav, and DDO all start from this same Reference adapter, which is also used as the frozen reference model. LoRA (Hu et al., 2022) is applied to the Q/K/V/O attention projections and MLP projections, with rank 3232, alpha 3232, and dropout 0.050.05. All preference runs use AdamW (Loshchilov and Hutter, 2019) with betas (0.9,0.999)(0.9,0.999), epsilon 10−810^-8, weight decay 0.00.0, gradient clipping at norm 1.01.0, and a linear learning-rate schedule. Unless otherwise stated, preference-trained methods use learning rate 5×10−65× 10^-6, warmup ratio 0.030.03, and β=0.1β=0.1. DDO uses α=0.5α=0.5. To construct qrefq_ref, the reference model scores the executable action field of each branch output. C.3 DTC Collection Cost Collector Requests Tokens Env. steps DTC 28.6 59.7K 48.3 Without DTC 65.3 156.0K 64.7 Table 11: Collection cost per unique successful trajectory on BabyAI and BabaIsAI. On BabyAI and BabaIsAI, DTC uses 44% of the expert requests, 38% of the expert tokens, and 75% of the environment steps used by Without DTC per unique successful trajectory. RTO requires no additional expert calls or environment interactions. C.4 Evaluation and Ablation Protocols A fixed seed protocol is used throughout. Training collection and evaluation use disjoint task seeds or WebShop evaluation sessions. BabyAI and BabaIsAI use the epoch-5 preference policies, while WebShop uses the epoch-15 preference policies. Main BabyAI and BabaIsAI results use sampled decoding with a temperature of 0.60.6, top-p of 0.950.95, and a max token limit of 81928192. WebShop coverage evaluation uses the same model and decoding/environment configuration as the corresponding success-rate evaluation. Main results. For BabyAI and BabaIsAI, success rate is computed from 50 rollouts per evaluation item. H-ESD and ESD use 20 rollouts for each of three seeds used for coverage evaluation and are averaged. For WebShop, success rate is reported over 50 evaluation sessions. WebShop coverage uses sessions 500500, 501501, and 502502, with 20 rollouts per session and varied LLM sampling seeds. Failed purchases, invalid actions, retry exhaustion, and max-step failures count toward the fixed rollout denominator. Rollouts interrupted by system failures or missing model outputs are excluded and replaced with reruns. Component comparison. The component comparison in Table 1 uses the BabyAI and BabaIsAI evaluation protocol described above. Settings without DTC train on same-state comparisons derived from separately sampled rollouts: successful and failed trajectories are sampled for the same task seed, converted into a preference comparison at their first shared decision state divergence, and matched to the corresponding budget for training comparisons. Settings with DTC use the state-aligned branch sets. All four settings use the corresponding DPO and DDO hyperparameters in this appendix and report benchmark averages. Strategic recovery. The strategic recovery probe in Figure 5 is run on BabyAI. For each method-task pair, we use successful evaluation rollouts, sample one interior decision point per rollout, replace the source action with a different valid task action, and roll out the same model from the edited prefix. To keep the probe balanced, the probe uses up to 30 successful rollouts per method-task pair. The reported value is success over valid edited rollouts. Imitation control. The imitation control in Table 3 is run on BabyAI and BabaIsAI. It starts from the same SFT initialization for each task and continues SFT on all successful DTC branches. The matched condition uses the same optimizer step count as DDO. The full-epoch condition runs successful-only imitation for the same nominal epoch count as DDO. Exposure is the number of processed training examples normalized by the DDO comparison exposure. Metrics use the two-benchmark aggregate. Decoding controls. The decoding diversification ablation in Table 4 reports an equal-weight benchmark macro average over BabyAI, BabaIsAI, and WebShop. The temperature sweep evaluates DPO at T∈0.0,0.1,0.6,1.0,1.5T∈\0.0,0.1,0.6,1.0,1.5\ with one rollout per item. The DDO setting uses sampled decoding at T=0.6T=0.6. All preference methods use epoch-5 policies for BabyAI and BabaIsAI and epoch-15 policies for WebShop. Coverage growth. Coverage growth curves use the BabyAI, BabaIsAI, and WebShop rollouts used for coverage evaluation. For each N, task averages are computed within each benchmark and then summarized with the three-benchmark aggregate. Branch-Set Target Distribution. The target distribution comparison in Table 7 uses β=0.1β=0.1. Under the parameterization in Eq. (3), α=0.0α=0.0 targets a uniform distribution over the observed successful branches, α=0.5α=0.5 contracts the reference log-odds among successful branches halfway toward uniformity, and α=1.0α=1.0 preserves the reference distribution restricted to successful branches. Margin sharpness sensitivity. The β sensitivity figure fixes α=0.5α=0.5 and varies β, using the two-benchmark aggregate. Appendix D Example LLM Calls This appendix shows representative input prompts and LLM outputs for each benchmark used in our evaluation. The three benchmarks share a common thought-action response format: the model produces a reasoning trace followed by an executable action, which a benchmark-specific parser extracts and forwards to the environment (Appendix B.1). Figures 9–12 illustrate this format on a single decision step in each benchmark. BabyAI. The input prompt in Figure 9 contains the task instruction, a textual rendering of the gridworld observation around the agent, the recent interaction history, and the admissible action set. The LLM output reasons over the visible objects and the current goal before emitting a single low-level navigation or manipulation action. BabaIsAI. The input prompt in Figure 10 additionally exposes the active rule configuration of the puzzle, since success depends on identifying and, when necessary, manipulating the rules. The model reasons over the grid state and active rules, then outputs either a goal-directed action or an action that changes the puzzle’s rule configuration. WebShop. The input prompt in Figure 12 provides the user’s purchase instruction, the contents of the current page, and the available interaction options such as search, product clicks, option selection, and purchase. The LLM output reasons about product attributes relative to the instruction before emitting a single page interaction. BabyAI Input prompt. ⬇ [user message 1] You are an agent playing a simple navigation game. Your goal is to go to the green box. The following are the possible actions you can take in the game, followed by a short description of each action: turn left: turn to the left, turn right: turn to the right, go forward: take one step forward, pick up: pick up the object directly in front of you (1 step forward), drop: place the object you are carrying on an empty tile directly in front of you (1 step forward), toggle: interact with the object directly in front of you; effects depend on object and state (1 step forward). In a moment I will present you an observation. Tips: - Use ’pick up’ to collect carryable items directly in front of you (1 step forward). - Use ’toggle’ to interact with the object directly in front of you. - Use ’drop’ to place the carried object on an empty tile directly in front of you (1 step forward). - Some actions may have no effect if their preconditions are not met (for example, blocked front tile or no carried object). - Avoid repeating the same action over and over if the observation doesn’t change. PLAY! [user message 2] Current Observation: a wall 6 steps forward a wall 3 steps left a green box 1 step right and 5 steps forward Your response should use the following format: Thought: <your thoughts> Action: <your next action> LLM output. ⬇ Thought: I need to reach the green box. The green box is 1 step right and 5 steps forward. But there is a wall 6 steps forward and a wall 3 steps left. I should first move forward to avoid the walls. Let’s try moving forward. Action: go forward Figure 9: Example BabyAI input prompt and LLM output. BabaIsAI Input prompt: task instructions. ⬇ [user message 1] BabaIsAI is a puzzle game where you can manipulate the rules of each level. The following are the possible actions you can take in the game, followed by a short description of each action: idle: wait for one step, up: take one step up, right: take one step to the right, down: take one step down, left: take one step to the left. Tips: - Examine the level carefully, noting all objects and text blocks present. - Identify the current rules, which are formed by text blocks in the format "[Subject] IS [Property]" (e.g. "BABA IS YOU"). - Consider how you can change or create new rules by moving text blocks around. - Remember that you can only move objects or text that are not defined as "STOP" or similar immovable properties. - Your goal is usually to reach an object defined as "WIN", but this can be changed. - Think creatively about how changing rules can alter the properties and behaviors of objects in unexpected ways. - If stuck, try breaking apart existing rules or forming completely new ones. - Sometimes the solution involves making yourself a different object or changing what counts as the win condition. PLAY! Figure 10: Example BabaIsAI input prompt (part 1). BabaIsAI (continued) Input prompt: current state and output format. ⬇ [user message 2] Current Observation: Active rules: ball is win wall is stop baba is you Objects on the map: wall 4 steps to the right and 4 step up rule ‘ball‘ 7 steps to the right and 4 step up rule ‘is‘ 8 steps to the right and 4 step up rule ‘win‘ 9 steps to the right and 4 step up rule ‘wall‘ 1 step to the left and 3 step up rule ‘is‘ 3 step up rule ‘stop‘ 1 step to the right and 3 step up wall 4 steps to the right and 3 step up ball 5 steps to the right and 3 step up wall 4 steps to the right and 2 step up wall 4 steps to the right and 1 step up wall 4 steps to the right wall 4 steps to the right and 1 step down rule ‘baba‘ 1 step to the left and 2 steps down rule ‘is‘ 2 steps down rule ‘you‘ 1 step to the right and 2 steps down wall 4 steps to the right and 2 steps down Your response should use the following format: Thought: <one short thought> Action: <exactly one of: idle, up, right, down, left> Rules: - The Action line must be exactly one of: idle, up, right, down, left. - Do not add any other words, punctuation, or explanation on the Action line. - Output exactly one Thought line and exactly one Action line. - Do not output anything after the Action line. LLM output. ⬇ Thought: I need to break the "WALL IS STOP" rule to pass through the wall barrier blocking the path to the ball. Action: left Figure 11: Example BabaIsAI input prompt and LLM output (part 2). WebShop Input prompt. ⬇ [user message 1] You are a sequential WebShop text-environment agent. Your mission is to satisfy the shopping instruction by navigating pages, selecting required options, and buying the best matching product. Valid action forms: - search[keywords]: search from the search page only. - click[value]: click a visible button, product id, navigation control, option, or click[buy now]. Choose exactly one valid action at each decision step. [user message 2] Current Observation: Instruction: [SEP] i am looking for oral hygiene dental tools of design: set of 4, and price lower than 40.00 dollars [SEP] Back to Search [SEP] < Prev [SEP] design [SEP] dental curette [SEP] dental probe [SEP] mouth mirror [SEP] set of 4 (dental hygiene kit) [SEP] tarter scraper [SEP] Oral Care Dental Tools, Professional Dental Picks for Home use, Dental Tarter Scraper for Tooth Stains Remover, Plaque Remover for Teeth, Dentist Pick Stainless Steel (Dental Probe) [SEP] Price: $4.99 [SEP] Rating: N.A. [SEP] Description [SEP] Features [SEP] Reviews [SEP] Buy Now Available actions for this state: - click[back to search] - click[< prev] - click[description] - click[features] - click[reviews] - click[buy now] - click[dental curette] - click[dental probe] - click[mouth mirror] - click[set of 4 (dental hygiene kit)] - click[tarter scraper] Output exactly two lines and nothing else: Thought: <your thoughts> Action: <one valid action> LLM output. ⬇ Thought: The current item is a dental probe, not a set of 4. I need to select the set option. Action: click[set of 4 (dental hygiene kit)] Figure 12: Example WebShop input prompt and LLM output.