Paper deep dive
Explicit Trajectory Diversity for RL-Based Post-Training of LLM Agents
Huaiyu Fu, Heng Cao, Hao Wang, Jian Ya, Tao Chen
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 10/4/2026, 3:08:08 AM
Summary
The paper introduces Trajectory-guided Joint Policy Optimization (TJPO), a framework for optimizing explicit, user-specified trajectory diversity in RL-based post-training of LLM agents. Unlike implicit diversity methods, TJPO uses task-specific trajectory descriptors to measure behavioral variation and converts group-level diversity signals into trajectory-level learning credits within a group-based policy optimization setting (specifically GiGPO). Experiments on Sokoban and ALFWorld demonstrate that TJPO improves task-specific diversity while maintaining competitive task performance, and enhances robustness to environmental changes and user constraints.
Entities (10)
Relation Signals (7)
TJPO → evaluatedon → Sokoban
confidence 98% · Experiments on Sokoban and ALFWorld show that TJPO improves task-specific trajectory diversity
TJPO → evaluatedon → AlfWorld
confidence 98% · Experiments on Sokoban and ALFWorld show that TJPO improves task-specific trajectory diversity
Huaiyu Fu → affiliatedwith → Microsoft
confidence 95% · Huaiyu Fu Affiliation: Microsoft
TJPO → optimizes → Trajectory Descriptors
confidence 95% · Our key idea is to define diversity through user-specified, task-specific trajectory descriptors... and then measure diversity as a set-level functional
Qwen2.5-7B-Instruct → usedin → AlfWorld
confidence 95% · We use Qwen2.5-7B-Instruct as the base model for ALFWorld
Qwen3-VL-4B-Instruct → usedin → Sokoban
confidence 95% · and Qwen3-VL-4B-Instruct for Sokoban.
TJPO → buildson → GiGPO
confidence 90% · We implement our method on top of GiGPO [13], a group-based RL algorithm for long-horizon LLM agent training.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:LLM agents often admit multiple high-quality solutions to the same task, differing in reasoning structure, tool-use pattern, or interaction trajectory. Yet existing notions of diversity in LLM post-training are mostly implicit, arising from general stochasticity and regularization mechanisms rather than explicitly targeting task-relevant behavioral variation. While such implicit diversity can be useful, it does not directly specify which forms of behavioral variation should be encouraged for a given task. In this work, we study explicit trajectory diversity in RL-based post-training for LLMs. Our key idea is to define diversity through user-specified, task-specific trajectory descriptors, which map each sampled trajectory to an interpretable behavioral representation, and then measure diversity as a set-level functional over the resulting descriptor matrix. Building on this formulation, we introduce Trajectory-guided Joint Policy Optimization(TJPO), a single-policy framework that optimizes explicit diversity over sampled trajectory groups, avoiding the need for population-based policy training, and instantiate it within group-based policy optimization through trajectory-level learning signals. This design makes the diversity objective both interpretable and controllable. Experiments on Sokoban and ALFWorld show that TJPO improves task-specific trajectory diversity while maintaining competitive task performance. Descriptor and trajectory analyses show that the learned variation follows the specified behavioral dimensions and includes distinct successful strategies. Extra experiment results suggest that explicitly shaping trajectory diversity can help LLM agents satisfy user requirements and remain effective when task conditions change.
Tags
Links
- Source: https://arxiv.org/abs/2609.38805v1
- Canonical: https://arxiv.org/abs/2609.38805v1
Trouble viewing inline? Open PDF directly →
Full Text
75,296 characters extracted from source content.
Expand or collapse full text
Explicit Trajectory Diversity for RL-Based Post-Training of LLM Agents Huaiyu Fu Affiliation: Microsoft Heng CaoHao WangJian YaoTao Chen Affiliation: tuyoogame Abstract Large language model (LLM) agents often admit multiple high-quality solutions to the same task, differing in reasoning structure, tool-use pattern, or interaction trajectory. Yet existing notions of diversity in LLM post-training are mostly implicit, arising from general stochasticity and regularization mechanisms rather than explicitly targeting task-relevant behavioral variation. While such implicit diversity can be useful, it does not directly specify which forms of behavioral variation should be encouraged for a given task. In this work, we study explicit trajectory diversity in RL-based post-training for LLMs. Our key idea is to define diversity through user-specified, task-specific trajectory descriptors, which map each sampled trajectory to an interpretable behavioral representation, and then measure diversity as a set-level functional over the resulting descriptor matrix. Building on this formulation, we introduce Trajectory-guided Joint Policy Optimization (TJPO), a single-policy framework that optimizes explicit diversity over sampled trajectory groups, avoiding the need for population-based policy training, and instantiate it within group-based policy optimization through trajectory-level learning signals. This design makes the diversity objective both interpretable and controllable: practitioners can specify whether diversity should correspond to action composition, location visitation patterns, spatial coverage, or other task-relevant behavioral dimensions. Experiments on Sokoban and ALFWorld show that TJPO improves task-specific trajectory diversity while maintaining competitive task performance. Descriptor and trajectory analyses show that the learned variation follows the specified behavioral dimensions and includes distinct successful strategies. Beyond diversity metrics, evaluations on an ALFWorld subset with verified alternative solutions demonstrate practical benefits without additional training: compared with GiGPO, TJPO improves compliant success under user-specified location constraints by 8.3 percentage points and task success under environmental disruptions by 7.5 percentage points. These findings suggest that explicitly shaping trajectory diversity can help LLM agents satisfy user requirements and remain effective when task conditions change. 1 Introduction Large language models (LLMs) are increasingly deployed as agents that solve multi-step tasks through planning, interaction, and tool use [10, 62]. In such settings, a single objective often admits multiple high-quality solutions that differ substantially in their behavioral patterns, such as plan decomposition, tool selection, and interaction trajectories [56, 41, 39]. For example, consider a household agent asked to place a target object in a desired location. One successful strategy may first explore the environment to identify relevant objects and receptacles, verify object states, and then execute the manipulation sequence. Another strategy may directly exploit known task regularities, navigate to likely object locations, and perform a shorter sequence of actions. Both strategies can solve the task, but they differ in object-interaction patterns, action composition, and robustness to changes in the environment. Similarly, in a planning game such as Sokoban, two successful solutions may reach the same goal state while following different routes, pushing boxes in different orders, or using different intermediate subgoals. Therefore, beyond average success rate or expected return, practical deployment also benefits from sufficient behavioral coverage: an agent should be able to produce a set of complementary high-quality strategies rather than relying on a single dominant path [2, 9]. Such coverage is useful because the preferred strategy may depend on deployment constraints. A conservative retrieve-then-verify workflow may be preferable when reliability is critical, while a more direct execution-oriented strategy may be preferable when latency or interaction cost is limited. Moreover, when the environment changes (e.g., when an object is unavailable, a tool fails, or the accessible action set is restricted), an agent with multiple behaviorally distinct strategies is more likely to retain a viable alternative path. This motivates diversity not merely as variation in final responses, but as task-relevant variation in the trajectories by which agents achieve their goals. Figure 1: Motivation and schematic overview of TJPO. Without an explicit diversity objective, successful rollouts may concentrate on similar behaviors (top). TJPO augments the group-based advantage with descriptor-based trajectory-level diversity credits to encourage task-relevant behavioral variation (bottom). However, such behavioral coverage does not follow automatically from RL-based post-training, where optimization primarily targets task rewards or preference alignment [1, 12, 3, 48, 13]. Without incentives to preserve diversity, policy optimization can favor a small set of reliably rewarded behaviors, while other valid strategies become less likely to be sampled and reinforced [23]. Existing mechanisms can encourage diversity through decoding heuristics, entropy regularization, or other distribution-level regularizers [53, 25]. While these mechanisms can support exploration and response variation, encouraging variation alone does not specify what should be diverse for a particular task. These mechanisms typically operate on generic properties of the policy distribution rather than task-specific behavioral dimensions, leaving their effect on strategy-level diversity implicit. Consequently, greater response variation does not necessarily translate into distinct reasoning structures, planning patterns, or tool-use behaviors that matter in practice. The resulting diversity may therefore remain weakly aligned with task requirements and difficult to interpret or control. This motivates a key question: how can we explicitly define and optimize task-relevant trajectory diversity in RL-based post-training while maintaining strong task performance? To address this question, we propose Trajectory-guided Joint Policy Optimization (TJPO), a simple framework for optimizing user-specified, task-specific trajectory diversity in RL-based post-training. Our key idea is to represent each sampled trajectory using a set of trajectory descriptors chosen to reflect the behavioral variations that matter for the target environment, and to define diversity as a set-level functional over the resulting descriptor matrix. This formulation makes the diversity objective explicit and controllable, rather than leaving diversity to emerge indirectly from generic stochasticity or response variation. We further integrate this signal into group-based policy optimization by translating group-level diversity into trajectory-level credit, making TJPO readily compatible with GRPO-style and GiGPO-style training. Figure 1 illustrates the motivation and diversity credit introduced by TJPO. We evaluate TJPO on two complementary environments: Sokoban, a planning-intensive game environment, and ALFWorld, a long-horizon agentic environment [41, 39]. For each environment, we instantiate user-specified, task-specific trajectory descriptors tailored to the corresponding notion of meaningful behavioral variation. On both environments, TJPO improves explicit trajectory diversity over strong group-based RL baselines while maintaining competitive task performance. These results suggest that task-specific trajectory descriptors provide a practical interface for optimizing meaningful behavioral diversity in RL-based post-training. Our contributions are summarized as follows: • We formulate explicit trajectory diversity for RL-based post-training through user-specified, task-specific trajectory descriptors and set-level diversity functionals, providing an interpretable and controllable way to specify what should be diverse. • We introduce Trajectory-guided Joint Policy Optimization (TJPO), a single-policy framework that converts group-level diversity into trajectory-level learning signals, enabling explicit diversity optimization within group-based RL without maintaining a population of policies. • We validate TJPO on ALFWorld and Sokoban, showing that it improves task-specific trajectory diversity while maintaining competitive task performance. Further trajectory-level and intervention-based evaluations show that the learned diversity corresponds to distinct successful strategies and can improve compliance with user constraints and robustness to environmental disruptions without additional training. 2 Preliminary 2.1 RL for LLMs In RL-based post-training for LLMs, generation is modeled as a sequential decision-making process. Given an input x∈x , the model acts as a policy that autoregressively generates a trajectory τ=(τ1,τ2,…,τT)∈τ=(τ^1,τ^2,…,τ^T) , where T denotes the space of all possible outputs. The policy is parameterized by θ and factorizes as πθ(τ∣x):=∏t=1Tπθ(τt∣x,τ<t), _θ(τ x):= _t=1^T _θ(τ^t x,τ^<t), (1) where τ<t=(τ1,…,τt−1)τ^<t=(τ^1,…,τ^t-1). A reward function is then used to evaluate the generated trajectory and optimize the policy. 2.2 Group-based Reinforcement Learning Algorithms Group Relative Policy Optimization (GRPO) GRPO eliminates the need for a separate critic model by estimating the baseline from a group of sampled trajectories. For each input x, it samples a group of trajectories τ1,…,τG\ _1,…, _G\ from the old policy πold _old. Writing the trajectory likelihood ratio as ρi(θ)=πθ(τi|x)/πold(τi|x) _i(θ)= _θ( _i|x)/ _old( _i|x), GRPO optimizes the current policy πθ _θ via [1G∑i=1G(min(ρi(θ)Ai,clip(ρi(θ),1−ϵ,1+ϵ)Ai)−βKL(πθ∥πref))],E [ 1G _i=1^G ( ( _i(θ)A_i,clip ( _i(θ),1-ε,1+ε )A_i )- _KL( _θ\| _ref) ) ], (2) where the expectation is over x∼x and τii=1G∼πold(⋅|x)\ _i\_i=1^G _old(·|x), ϵε and β are hyperparameters, and KL(πθ∥πref)=πref(τi|x)πθ(τi|x)−logπref(τi|x)πθ(τi|x)−1,Ai=ri−mean(r1,…,rG)std(r1,…,rG).D_KL( _θ\| _ref)= _ref( _i|x) _θ( _i|x)- _ref( _i|x) _θ( _i|x)-1, A_i= r_i-mean(\r_1,…,r_G\)std(\r_1,…,r_G\). (3) Group-in-Group Policy Optimization (GiGPO) While GRPO is effective for single-turn generation, long-horizon agent tasks present a more challenging credit assignment problem due to sparse and delayed rewards. GiGPO extends group-based RL by introducing an additional step-level relative advantage on top of the trajectory-level advantage used in GRPO. To provide finer-grained supervision, GiGPO constructs step-level groups using anchor states, i.e., repeated or aligned environment states observed across trajectories. For an action taken at step t in trajectory i, it computes a micro advantage by comparing it against other actions originating from the same anchor-state group: Ai,tmicro=ui,t−mean(ℋ(si,t))std(ℋ(si,t)),A_i,t^micro= u_i,t-mean(H(s_i,t))std(H(s_i,t)), (4) where ℋ(si,t)H(s_i,t) denotes the set of step-level utilities associated with the anchor-state group of si,ts_i,t. The final advantage combines the trajectory-level signal from GRPO with the step-level signal: A^i,t=Ai+ωAi,tmicro, A_i,t=A_i+ω A_i,t^micro, (5) where ω controls the strength of the micro-level credit assignment. GiGPO is particularly suitable for long-horizon LLM agent training, where useful supervision must be propagated not only across full trajectories but also across individual decision steps. 3 Method We study explicit trajectory diversity in group-based policy optimization for LLMs. Our goal is to encourage intent-aligned behavioral variation, namely, variation along task-relevant dimensions specified by the practitioner, rather than generic randomness induced by token-level entropy or decoding noise. 3.1 Problem Setup Following the notation in Section 2.2, for each input x we sample a group of trajectories (x)=τii=1G,τi∼πθ(⋅∣x).G(x)=\ _i\_i=1^G, _i _θ(· x). (6) Each trajectory τi _i receives a task reward ritask=R(x,τi)r_i^task=R(x, _i). Our goal is to augment this reward with an explicit diversity signal defined over the sampled trajectory group (x)G(x). 3.2 Intent-Aligned Trajectory Descriptors Definition 1 (Intent-Aligned Trajectory Descriptor). Given the trajectory space T, an intent-aligned trajectory descriptor is a user-specified scalar function ϕk:→ℝ,k=1,…,m, _k:T , k=1,…,m, (7) that measures one task-relevant axis of behavioral variation. A descriptor family ϕkk=1m\ _k\_k=1^m induces a trajectory-to-vector mapping (τ)=[ϕ1(τ),…,ϕm(τ)]⊤∈ℝm.z(τ)= [ _1(τ),…, _m(τ) ] ^m. (8) For each sampled trajectory τi∈(x) _i (x), we write i=(τi)z_i=z( _i). Collecting all descriptors in the group yields the descriptor matrix Z=[1⋯G]⊤∈ℝG×m.Z=[z_1·sz_G] ^G× m. (9) By construction, the descriptor family ϕkk=1m\ _k\_k=1^m is intent-aligned: it explicitly specifies which forms of trajectory variation are desirable for the task, rather than relying on task-agnostic statistics over the policy’s state-action distribution. 3.3 Group-Level Explicit Diversity We now define explicit diversity as a set-level function over the descriptor vectors induced by a sampled trajectory group. Definition 2 (Explicit Diversity Functional). Given a descriptor matrix Z∈ℝG×mZ ^G× m, where each row corresponds to the descriptor vector of one sampled trajectory, an explicit diversity functional is a mapping :ℝG×m→ℝ,D:R^G× m , (10) which assigns a scalar diversity score to the trajectory group. We require D to be permutation-invariant with respect to the ordering of trajectories, i.e., for any G×G× G permutation matrix P, (Z)=(PZ).D(Z)=D(PZ). (11) Pairwise diversity. A simple choice is the average pairwise dispersion: pair(Z)=2G(G−1)∑1≤i<j≤Gd(i,j),D_pair(Z)= 2G(G-1) _1≤ i<j≤ Gd(z_i,z_j), (12) where d(⋅,⋅)d(·,·) is a distance function, e.g., Euclidean distance or cosine distance. By construction, pairD_pair is permutation-invariant. Determinantal diversity. To capture set-level coverage beyond pairwise comparisons, we also consider a determinantal form. Let Kij=k(i,j),K_ij=k(z_i,z_j), (13) where k(⋅,⋅)k(·,·) is a positive semidefinite kernel, such as an RBF kernel. We define det(Z)=logdet(K+ηI),D_det(Z)= (K+η I), (14) where η>0η>0 ensures numerical stability. Maximizing (14) encourages the sampled trajectories to span a larger volume in the descriptor space, thereby reducing redundancy. This functional is also permutation-invariant, since permuting the rows of Z only induces a simultaneous row-column permutation of K, which leaves the determinant unchanged. The formulation above is intentionally modular: any permutation-invariant functional over the descriptor set can be plugged into our framework. 3.4 From Group Diversity to Trajectory-Level Credit A subtle but important issue arises when combining explicit diversity with GRPO-style training. Since GRPO computes group-relative advantages, adding the same group-level scalar bonus to every trajectory has no effect after baseline subtraction. Indeed, the shared term in r¯i=ritask+λ(Z)∀i, r_i=r_i^task+ (Z) ∀ i, (15) vanishes after group centering, so diversity must be converted into trajectory-level credit. For pairwise diversity, we use the average distance to the remaining group members: cipair(Z)=1G−1∑j≠id(i,j).c_i^pair(Z)= 1G-1 _j≠ id(z_i,z_j). (16) As derived in Appendix A, this credit is a positive affine transformation of the exact leave-one-out change within a fixed group and therefore yields identical group-standardized values. For determinantal diversity, with A=K+ηIA=K+η I, we use cidet(Z)=det(A)det(A−i,−i)=exp(det(Z)−det(Z−i)).c_i^det(Z)= (A) (A_-i,-i)= \! (D_det(Z)-D_det(Z_-i) ). (17) This is the exponential of the leave-one-out log-determinant change. Finally, we standardize the credits within each sampled group and add them to the GiGPO advantage: c^i c_i =ci−c¯maxstd(c),ϵ, = c_i- c \std(c),ε\, (18) A~i,t A_i,t =A^i,t+λc^i, = A_i,t+λ c_i, (19) where ϵ>0ε>0 handles near-zero variance and λ≥0λ≥ 0 controls the task–diversity trade-off. The trajectory-level credit is shared across the valid response tokens associated with trajectory i. Our method preserves the simplicity of GRPO-style training while explicitly optimizing task-relevant trajectory diversity. The key difference from implicit diversity regularizers is that diversity is defined over full-trajectory descriptors chosen to match the intended behavioral variation of each environment, rather than generic statistics of token-level stochasticity. Taken together, our framework cleanly separates what should be diverse from how diversity is optimized. The descriptor family ϕk\ _k\ specifies the intended axes of variation, while the diversity functional D specifies how a group of sampled trajectories should spread out in the resulting descriptor space. This modular design allows the same optimization framework to be instantiated across diverse environments with task-specific notions of meaningful diversity. We term our method Trajectory-guided Joint Policy Optimization (TJPO). 4 Experiments We evaluate TJPO through the following questions: Q1. Can TJPO improve task-specific diversity while maintaining strong task performance? Q2. How do trajectory descriptors and diversity functionals shape the learned behavioral variation? Q3. How does the diversity coefficient λ affect the diversity–performance trade-off? Q4. Does the resulting behavioral diversity provide practical benefits under user constraints and environmental disruptions? 4.1 Experimental Setup Environments. We evaluate on ALFWorld [41] and Sokoban [39]. ALFWorld requires an agent to complete household manipulation tasks through multi-step text interactions, where trajectories consist of observations, reasoning, and textual actions. Sokoban is a grid-based box-pushing puzzle in which the agent must plan action sequences to push boxes onto target cells. We provide detailed environment descriptions in Appendix C.1. Models. We use Qwen2.5-7B-Instruct as the base model for ALFWorld and Qwen3-VL-4B-Instruct for Sokoban. Within each environment, all trained policies start from the same base model and use the same rollout budget, optimization hyperparameters, and evaluation protocol. Our decoding-only baseline reuses the trained GiGPO checkpoint without any additional optimization. Training backbone. We implement our method on top of GiGPO [13], a group-based RL algorithm for long-horizon LLM agent training. For each input instance, the policy samples a group of trajectories. We compute a trajectory-level diversity credit for each trajectory, standardize the credits within the sampled group, and add the resulting signal to the GiGPO advantage as described in Equation 19. Implementation details are provided in Appendix C.2. Evaluation metrics. We report both task performance and explicit diversity. For task performance, we use success rate and average trajectory reward. For diversity, we sample NevalN_eval trajectories per input and compute descriptor-based diversity over the successful subset of sampled trajectories using the corresponding evaluation descriptor. Baselines. Our primary baseline is GiGPO without explicit diversity optimization. For ALFWorld, we also evaluate GiGPO+P and GiGPO+RP, inference-only baselines that apply a presence penalty of 0.50.5 or repetition penalty of 1.051.05 to the same checkpoint while leaving all other evaluation settings unchanged. We compare action-type and location descriptors with pairwise or determinantal diversity. For Sokoban, we compare GiGPO with two task-specific descriptors, denoted as direction-transition and spatial-coverage, which are detailed in Appendix D. 4.2 Main Results on ALFWorld Table 1: Pairwise diversity on ALFWorld. Method Success ↑ D_act^pair ↑ D_loc^pair ↑ GiGPO 0.9160 0.0495 0.0858 GiGPO + P 0.9082 0.0458 0.0895 GiGPO + RP 0.9229 0.0494 0.0873 TJPOpairact_act^pair 0.8682 0.0780 0.1002 TJPOpairloc_loc^pair 0.9053 0.0601 0.1316 Table 2: DPP diversity (×1000× 1000) on ALFWorld. Method Success ↑ D_act^dpp ↑ D_loc^dpp ↑ GiGPO 0.9160 1.887 2.500 GiGPO + P 0.9082 1.858 2.063 GiGPO + RP 0.9229 1.135 1.544 TJPOdppact_act^dpp 0.9160 4.100 6.655 TJPOdpploc_loc^dpp 0.8779 3.058 7.940 Tables 2 and 2 report the pairwise and determinantal diversity results on ALFWorld, respectively. All TJPO variants use λ=0.3λ=0.3. For determinantal diversity DdppD^dpp, we use the determinant-root score defined in Appendix E, with values reported in units of 10−310^-3. We compare standard GiGPO, two decoding-only penalty baselines, and four TJPO variants obtained by combining two task-specific trajectory descriptors, action type and location, with either pairwise or determinantal diversity. The results show that TJPO improves explicit trajectory diversity under the intended descriptor metrics while maintaining competitive task performance. For the action-type descriptor, TJPO with pairwise diversity increases DactpairD_act^pair from 0.04950.0495 to 0.07800.0780, with a success rate of 0.86820.8682; the determinantal variant increases DactdppD_act^dpp from 1.887×10−31.887× 10^-3 to 4.100×10−34.100× 10^-3 with a success rate of 0.91600.9160. For the location descriptor, TJPO with pairwise diversity increases DlocpairD_loc^pair from 0.08580.0858 to 0.13160.1316, with a success rate of 0.90530.9053; the determinantal variant increases DlocdppD_loc^dpp from 2.500×10−32.500× 10^-3 to 7.940×10−37.940× 10^-3, with a success rate of 0.87790.8779. These results indicate that user-specified descriptors provide a controllable interface for shaping which aspects of trajectory behavior become diverse. The cross-metric results show that the descriptor determines which aspects of behavior become more diverse. Action-type optimization primarily changes the composition of action categories, whereas location-level optimization encourages broader variation in location visitation patterns. (a) GiGPO navigation strategies. (b) TJPO navigation strategies. Figure 2: Successful navigation strategies for “put a spraybottle in cabinet”. TJPO uses the location descriptor with pairwise diversity and λ=0.3λ=0.3. Both methods succeed in all eight rollouts. Nodes denote visited locations; edge labels and widths encode navigation transition counts across the eight rollouts. 4.3 Main Results on Sokoban Table 3: Main results on Sokoban. Method Success ↑ D_dir^pair ↑ D_spa^pair ↑ GiGPO 0.8281 0.0146 0.0092 TJPOpairdir_dir^pair 0.8428 0.0675 0.0503 TJPOpairspa_spa^pair 0.8057 0.0595 0.0518 Table 4: Ablation on the λ on ALFWorld. Method λ Success ↑ DlocpairD_loc^pair ↑ Avg. Steps ↓ TJPO 0 0.9160 0.0858 9.2217 TJPO 0.1 0.9209 0.0942 9.5811 TJPO 0.3 0.9053 0.1316 9.6947 TJPO 0.5 0.9004 0.1356 8.1150 Table 4 reports the main results on Sokoban. All TJPO variants use λ=0.3λ=0.3 and pairwise diversity. Compared with the GiGPO baseline, TJPO improves explicit trajectory diversity under both task-specific descriptors. When optimizing the direction-transition descriptor, TJPO increases DdirpairD_dir^pair from 0.01460.0146 to 0.06750.0675, with a success rate of 0.84280.8428. When optimizing the spatial-coverage descriptor, TJPO increases DspapairD_spa^pair from 0.00920.0092 to 0.05180.0518, with a success rate of 0.80570.8057. The results show that TJPO can substantially improve planning-level trajectory diversity while maintaining competitive task performance. Together with the ALFWorld results in Section 4.2, these findings suggest that our framework is not limited to tool-use trajectories, but also applies to planning-intensive game environments where meaningful diversity is reflected in movement patterns and state-space coverage. 4.4 Case Study Figure 2 compares the successful navigation strategies of GiGPO and TJPO for the task “put a spraybottle in cabinet”. GiGPO follows the same route in seven of eight rollouts. TJPO instead distributes its rollouts across three successful navigation strategies, including a more direct route through the Toilet. This example illustrates that explicit location descriptor encourages alternative efficient solution modes rather than merely increasing a scalar diversity score. We further expand the route-level case study in Appendix F. 4.5 Ablation on Diversity Weight Table 4 studies the effect of the diversity coefficient λ. For the location descriptor, increasing λ improves DlocpairD_loc^pair, from 0.08580.0858 at λ=0λ=0 to 0.13560.1356 at λ=0.5λ=0.5, while maintaining competitive task performance. The mean successful-trajectory length remains comparable at λ≤0.3λ≤ 0.3 and decreases to 8.11508.1150 steps at λ=0.5λ=0.5, indicating that the diversity gain does not result from longer successful interactions. 4.6 Practical Benefits of Behavioral Diversity Beyond diversity scores, we examine whether behavioral diversity helps agents satisfy user-specific requirements and cope with environmental disruptions. We use location access in ALFWorld as a concrete example: User Constraints specifies locations the agent must avoid, whereas Disruption makes task-relevant locations unavailable. Standard denotes the original task without either intervention. We compare GiGPO with location-pairwise TJPO at λ=0.5λ=0.5 on tasks with verified alternative solutions. Details are provided in Appendix G. Table 5: Practical benefits of behavioral diversity on ALFWorld. Method Standard Success ↑ User Constraints Success ↑ Disruption Success ↑ GiGPO 0.9917 0.7708 0.7792 TJPO 0.9750 0.8542 0.8542 Table 5 shows that TJPO improves compliant success under user constraints from 77.08%77.08\% to 85.42%85.42\% and task success under environmental disruption from 77.92%77.92\% to 85.42%85.42\%. These results illustrate the practical value of behavioral diversity in this setting: producing successful behaviors that meet user requirements and maintaining task completion when the environment is disrupted. 5 Related work Diversity in RL Prior work on policy diversity in deep reinforcement learning can be broadly organized according to the role that diversity plays in the learning process [50]. One line of work leverages diversity to facilitate exploration, where diverse behaviors are encouraged mainly as an intermediate mechanism for improving final task performance [19, 11, 35, 8, 36, 54]. A second line incorporates diversity through constrained optimization, either by maximizing task quality under diversity constraints or by maximizing diversity while enforcing quality constraints [32, 66, 14, 60, 70, 67, 68]. A third line jointly optimizes quality and diversity, commonly referred to as Quality-Diversity RL [7, 37, 44, 2, 51]. Our work extends the second paradigm to RL for LLM training. Diversity in LLMs Prior work has explored diversity in LLMs from several perspectives, including supervised fine-tuning, preference optimization, RL-based post-training, diversity evaluation. In supervised and preference-based post-training, existing methods aim to preserve or improve response diversity during alignment, often by modifying the training objective, constructing diverse preference pairs, or encouraging diverse high-quality generations [30, 24, 6, 4, 45]. Complementary studies evaluate linguistic, semantic, and epistemic diversity in LLM outputs, showing that diversity is an important dimension beyond average quality or accuracy [42, 16, 49]. In RL-related settings, recent work has begun to study diversity in LLM reasoning, while other approaches use divergence choices or entropy-based objectives to indirectly preserve exploration and mitigate diversity collapse [53, 25]. These works are closely related to ours, but they typically define or encourage diversity through variation among token-level diversity, embedding-based semantic diversity, or implicit distributional regularization. In contrast, our work focuses on explicit trajectory diversity: diversity is defined through user-specified, task-specific trajectory descriptors that directly encode the intended behavioral variations of each environment, such as reasoning structures, planning patterns, or tool-use trajectories, and is optimized within RL-based post-training. 6 Conclusion, Limitations, and Discussion We study explicit trajectory diversity in RL-based post-training for LLMs. Unlike implicit notions of diversity that arise from sampling randomness, entropy regularization, or distributional constraints, our framework defines diversity through user-specified, task-specific trajectory descriptors. These descriptors make explicit which behavioral dimensions should vary for a given environment, such as planning patterns and tool-use trajectories. Building on this formulation, we introduce a framework to optimize diversity over sampled trajectory groups. By converting set-level diversity into trajectory-level learning signals, the proposed method can be integrated into group-based RL training, such as GiGPO. Across Sokoban and agentic tasks, our experiments show that explicit trajectory diversity can be consistently improved while maintaining strong task performance. While our results are encouraging, several aspects merit further study. First, our framework relies on task-specific trajectory descriptors. This design is intentional, since explicit diversity should reflect the behavioral variations that matter for the target environment. At the same time, different descriptor choices may emphasize different aspects of behavior, and developing better interfaces or reusable descriptor templates across task families, such as reasoning, planning, tool use, and embodied interaction, would make the framework easier to apply while preserving its user-controlled nature. Second, although our experiments span planning and agentic tasks, broader evaluation on richer agent settings would further clarify the generality of explicit trajectory diversity. Promising directions include larger-scale tool ecosystems, open-ended web environments, multi-agent settings, longer horizons, larger action spaces, and more complex environmental shifts. Overall, our results suggest that explicitly defining and optimizing task-aligned trajectory diversity is a useful step toward more controllable, robust, and adaptable LLM agents. References [1] H. Bai, Y. Zhou, M. Cemri, J. Pan, A. Suhr, S. Levine, and A. Kumar (2024) Digirl: training in-the-wild device-control agents with autonomous reinforcement learning. Advances in Neural Information Processing Systems 37, p. 12461–12495. Cited by: Appendix B, §1. [2] S. Batra, B. Tjanaka, M. C. Fontaine, A. Petrenko, S. Nikolaidis, and G. Sukhatme (2023) Proximal policy gradient arborescence for quality diversity reinforcement learning. arXiv preprint arXiv:2305.13795. Cited by: §1, §5. [3] K. Chen, M. Cusumano-Towner, B. Huval, A. Petrenko, J. Hamburger, V. Koltun, and P. Krähenbühl (2025) Reinforcement learning for long-horizon interactive llm agents. arXiv preprint arXiv:2502.01600. Cited by: Appendix B, §1. [4] Y. Chen, S. Chakraborty, L. Wolf, Y. Paschalidis, and A. Pacchiano (2025) Post-training large language models for diverse high-quality responses. arXiv preprint arXiv:2509.04784. Cited by: §5. [5] X. Chu, H. Huang, X. Zhang, F. Wei, and Y. Wang (2025) GPG: a simple and strong reinforcement learning baseline for model reasoning. arXiv preprint arXiv:2504.02546. Cited by: Appendix B. [6] J. J. Y. Chung, V. Padmakumar, M. Roemmele, Y. Sun, and M. Kreminski (2025) Modifying large language model post-training for diverse creative writing. arXiv preprint arXiv:2503.17126. Cited by: §5. [7] G. Cideron, T. Pierrot, N. Perrin, K. Beguir, and O. Sigaud (2020) QD-rl: efficient mixing of quality and diversity in reinforcement learning. corr abs/2006.08505 (2020). arXiv preprint arXiv:2006.08505. Cited by: §5. [8] E. Conti, V. Madhavan, F. Petroski Such, J. Lehman, K. Stanley, and J. Clune (2018) Improving exploration in evolution strategies for deep reinforcement learning via a population of novelty-seeking agents. Advances in neural information processing systems 31. Cited by: §5. [9] A. Cully and Y. Demiris (2017) Quality and diversity optimization: a unifying modular framework. IEEE Transactions on Evolutionary Computation 22 (2), p. 245–259. Cited by: §1. [10] S. Du, J. Zhao, J. Shi, Z. Xie, X. Jiang, Y. Bai, and L. He (2026) A survey on the optimization of large language model-based agents. ACM Comput. Surv. 58 (9), p. 223:1–223:37. External Links: Link, Document Cited by: §1. [11] B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine (2018) Diversity is all you need: learning skills without a reward function. arXiv preprint arXiv:1802.06070. Cited by: §5. [12] L. Feng, W. Tan, Z. Lyu, L. Zheng, H. Xu, M. Yan, F. Huang, and B. An (2025) Towards efficient online tuning of vlm agents via counterfactual soft reinforcement learning. arXiv preprint arXiv:2505.03792. Cited by: Appendix B, §1. [13] L. Feng, Z. Xue, T. Liu, and B. An (2025) Group-in-group policy optimization for llm agent training, 2025. arXiv preprint arXiv:2505.10978. Cited by: Appendix B, §C.2, §1, §4.1. [14] M. Ghasemi, E. S. Crafts, B. Zhao, and U. Topcu (2021) Multiple plans are better than one: diverse stochastic planning. In Proceedings of the International Conference on Automated Planning and Scheduling, Vol. 31, p. 140–148. Cited by: §5. [15] D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. (2025) Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. Cited by: Appendix B. [16] Y. Guo, G. Shang, and C. Clavel (2025) Benchmarking linguistic diversity of large language models. Transactions of the Association for Computational Linguistics 13, p. 1507–1526. Cited by: §5. [17] Z. He, T. Liang, J. Xu, Q. Liu, X. Chen, Y. Wang, L. Song, D. Yu, Z. Liang, W. Wang, et al. (2025) DeepMath-103k: a large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning. arXiv preprint arXiv:2504.11456. Cited by: Appendix B. [18] A. Hochlehnert, H. Bhatnagar, V. Udandarao, A. Prabhu, and M. Bethge (2025) CuratedThoughts: data curation for rl training datasets. External Links: Link Cited by: Appendix B. [19] Z. Hong, T. Shann, S. Su, Y. Chang, T. Fu, and C. Lee (2018) Diversity-driven exploration strategy for deep reinforcement learning. Advances in neural information processing systems 31. Cited by: §5. [20] J. Hu (2025) Reinforce++: a simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.03262. Cited by: Appendix B. [21] J. Hu, Y. Zhang, Q. Han, D. Jiang, X. Zhang, and H. Shum (2025) Open-reasoner-zero: an open source approach to scaling up reinforcement learning on the base model. External Links: 2503.24290, Link Cited by: Appendix B. [22] A. Kazemnejad, M. Aghajohari, E. Portelance, A. Sordoni, S. Reddy, A. Courville, and N. L. Roux (2024) Vineppo: unlocking rl potential for llm reasoning through refined credit assignment. arXiv preprint arXiv:2410.01679. Cited by: Appendix B. [23] R. Kirk, I. Mediratta, C. Nalmpantis, J. Luketina, E. Hambro, E. Grefenstette, and R. Raileanu (2024) Understanding the effects of RLHF on LLM generalisation and diversity. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, External Links: Link Cited by: §1. [24] J. Lanchantin, A. Chen, S. Dhuliawala, P. Yu, J. Weston, S. Sukhbaatar, and I. Kulikov (2025) Diverse preference optimization. arXiv preprint arXiv:2501.18101. Cited by: §5. [25] L. Li, Z. Zhou, J. Hao, J. K. Liu, Y. Miao, W. Pang, X. Tan, W. Chu, Z. Wang, S. Pan, et al. (2025) The choice of divergence: a neglected key to mitigating diversity collapse in reinforcement learning with verifiable reward. arXiv preprint arXiv:2509.07430. Cited by: §1, §5. [26] X. Li, H. Zou, and P. Liu (2025) Limr: less is more for rl scaling. arXiv preprint arXiv:2502.11886. Cited by: Appendix B. [27] Z. Li, H. Rahmani, Q. Ke, and J. Liu (2025) LongDiff: training-free long video generation in one go. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), p. 17789–17798. Cited by: Appendix B. [28] Z. Li, H. Rahmani, J. Zhang, Y. Xue, M. Mirmehdi, J. Kuen, J. Gu, and J. Liu (2026) DiffGraph: an automated agent-driven model merging framework for in-the-wild text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Cited by: Appendix B. [29] Z. Li, J. Zhang, P. Hu, J. Kuen, J. Gu, H. Rahmani, and J. Liu (2026) Automatic method illustration generation for ai scientific papers via drawing middleware creation, evolution, and orchestration. In European Conference on Computer Vision, Cited by: Appendix B. [30] Z. Li, C. Chen, T. Xu, Z. Qin, J. Xiao, Z. Luo, and R. Sun (2025) Preserving diversity in supervised fine-tuning of large language models. In The Thirteenth International Conference on Learning Representations, Cited by: §5. [31] Z. Liu, C. Chen, W. Li, P. Qi, T. Pang, C. Du, W. S. Lee, and M. Lin (2025) Understanding r1-zero-like training: a critical perspective. arXiv preprint arXiv:2503.20783. Cited by: Appendix B. [32] M. A. Masood and F. Doshi-Velez (2019) Diversity-inducing policy gradient: using maximum mean discrepancy to find a set of diverse policies. arXiv preprint arXiv:1906.00088. Cited by: §5. [33] C. Meng, S. Song, H. Tong, M. Pan, and Y. Yu (2023) Deepscaler: holistic autoscaling for microservices based on spatiotemporal gnn with adaptive graph learning. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), p. 53–65. Cited by: Appendix B. [34] OpenAI (2024) Learning to reason with llms. Note: https://openai.com/index/learning-to-reason-with-llms Cited by: Appendix B. [35] J. Parker-Holder, A. Pacchiano, K. M. Choromanski, and S. J. Roberts (2020) Effective diversity in population based reinforcement learning. Advances in Neural Information Processing Systems 33, p. 18050–18062. Cited by: §5. [36] Z. Peng, H. Sun, and B. Zhou (2020) Non-local policy optimization via diversity-regularized collaborative exploration. arXiv preprint arXiv:2006.07781. Cited by: §5. [37] T. Pierrot, V. Macé, F. Chalumeau, A. Flajolet, G. Cideron, K. Beguir, A. Cully, O. Sigaud, and N. Perrin-Gilbert (2022) Diversity policy gradient for sample efficient quality-diversity optimization. In Proceedings of the Genetic and Evolutionary Computation Conference, p. 1075–1083. Cited by: §5. [38] P. Putta, E. Mills, N. Garg, S. Motwani, C. Finn, D. Garg, and R. Rafailov (2024) Agent q: advanced reasoning and learning for autonomous ai agents. arXiv preprint arXiv:2408.07199. Cited by: Appendix B. [39] M. B. Schrader (2018) Gym-sokoban. Note: https://github.com/mpSchrader/gym-sokobanGitHub repository Cited by: §C.1, §1, §1, §4.1. [40] G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu (2025) Hybridflow: a flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, p. 1279–1297. Cited by: §C.2. [41] M. Shridhar, X. Yuan, M. Côté, Y. Bisk, A. Trischler, and M. Hausknecht (2021) ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. In Proceedings of the International Conference on Learning Representations (ICLR), External Links: Link Cited by: §C.1, §1, §1, §4.1. [42] A. Shypula, S. Li, B. Zhang, V. Padmakumar, K. Yin, and O. Bastani (2025) Evaluating the diversity and quality of llm generated content. arXiv preprint arXiv:2504.12522. Cited by: §5. [43] K. Team, A. Du, B. Gao, B. Xing, C. Jiang, C. Chen, C. Li, C. Xiao, C. Du, C. Liao, et al. (2025) Kimi k1.5: scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599. Cited by: Appendix B. [44] B. Tjanaka, M. C. Fontaine, J. Togelius, and S. Nikolaidis (2022) Approximating gradients for differentiable quality diversity in reinforcement learning. In Proceedings of the Genetic and Evolutionary Computation Conference, p. 1102–1111. Cited by: §5. [45] H. Wang, H. Gu, H. Piao, K. Gong, Y. Ye, X. Yue, S. Han, Y. Guo, and D. Wu (2026) Learning while staying curious: entropy-preserving supervised fine-tuning via adaptive self-distillation for large reasoning models. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 13567–13581. Cited by: §5. [46] H. Wang, Z. Wang, K. Wu, J. Yao, Y. Ye, Y. Luo, D. Wu, J. Chen, and H. Yang (2026) CasePlay: self-play reinforcement learning from case reports for medical reasoning. In The Fortieth Annual Conference on Neural Information Processing Systems, Cited by: Appendix B. [47] Z. Wang, H. Wang, S. Feng, X. Yang, D. Wang, Y. Zhang, J. Lin, X. Ji, and H. Yang (2026) Deepmed: building a medical deepresearch agent via multi-hop med-search data and turn-controlled agentic training & inference. In Findings of the Association for Computational Linguistics: ACL 2026, p. 18160–18178. Cited by: Appendix B. [48] Z. Wang, K. Wang, Q. Wang, P. Zhang, L. Li, Z. Yang, X. Jin, K. Yu, M. N. Nguyen, L. Liu, et al. (2025) Ragen: understanding self-evolution in llm agents via multi-turn reinforcement learning. arXiv preprint arXiv:2504.20073. Cited by: Appendix B, §1. [49] D. Wright, S. Masud, J. Moore, S. Yadav, M. Antoniak, P. E. Christensen, C. Y. Park, and I. Augenstein (2025) Epistemic diversity and knowledge collapse in large language models. arXiv preprint arXiv:2510.04226. Cited by: §5. [50] S. Wu, J. Yao, H. Fu, Y. Tian, C. Qian, Y. Yang, Q. Fu, and Y. Wei (2023) Quality-similar diversity via population based reinforcement learning. In The eleventh international conference on learning representations, Cited by: §5. [51] H. Yang, J. Yao, W. Liu, Q. Wang, H. Qin, K. Tang, J. Xiong, C. Yu, K. Li, J. Xing, et al. (2025) Diverse policies recovering via pointwise mutual information weighted imitation learning. In International Conference on Learning Representations, Vol. 2025, p. 24860–24878. Cited by: §5. [52] J. Yao, R. Cheng, and K. C. Tan (2025) Var-math: probing true mathematical reasoning in llms via symbolic multi-instance benchmarks. arXiv preprint arXiv:2507.12885. Cited by: Appendix B. [53] J. Yao, R. Cheng, X. Wu, J. Wu, and K. C. Tan (2025) Diversity-aware policy optimization for large language model reasoning. arXiv preprint arXiv:2505.23433. Cited by: §1, §5. [54] J. Yao, W. Liu, H. Fu, Y. Yang, S. McAleer, Q. Fu, and W. Yang (2023) Policy space diversity for non-transitive games. Advances in Neural Information Processing Systems 36, p. 67771–67793. Cited by: §5. [55] J. Yao, X. Luo, R. Cheng, and K. C. Tan (2026) SLAT: segment-level adaptive trimming for efficient cot reasoning. arXiv preprint arXiv:2605.30832. Cited by: Appendix B. [56] S. Yao, H. Chen, J. Yang, and K. Narasimhan (2022) Webshop: towards scalable real-world web interaction with grounded language agents. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 35, p. 20744–20757. Cited by: §1. [57] Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, T. Fan, G. Liu, L. Liu, X. Liu, et al. (2025) Dapo: an open-source llm reinforcement learning system at scale, 2025. URL https://arxiv. org/abs/2503.14476. Cited by: Appendix B. [58] Y. Yuan, Q. Yu, X. Zuo, R. Zhu, W. Xu, J. Chen, C. Wang, T. Fan, Z. Du, X. Wei, et al. (2025) VAPO: efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:2504.05118. Cited by: Appendix B. [59] Y. Yuan, Y. Yue, R. Zhu, T. Fan, and L. Yan (2025) What’s behind ppo’s collapse in long-cot? value optimization holds the secret. arXiv preprint arXiv:2503.01491. Cited by: Appendix B. [60] T. Zahavy, B. O’Donoghue, A. Barreto, V. Mnih, S. Flennerhag, and S. Singh (2021) Discovering diverse nearly optimal policies with successor features. arXiv preprint arXiv:2106.00669. Cited by: §5. [61] W. Zeng, Y. Huang, W. Liu, K. He, Q. Liu, Z. Ma, and J. He (2025) 7B model and 8k examples: emerging reasoning with reinforcement learning is both effective and efficient. Note: https://hkust-nlp.notion.site/simplerl-reasonNotion Blog Cited by: Appendix B. [62] G. Zhang, H. Geng, X. Yu, Z. Yin, Z. Zhang, Z. Tan, H. Zhou, Z. Li, X. Xue, Y. Li, Y. Zhou, Y. Chen, C. Zhang, Y. Fan, Z. Wang, S. Huang, F. P. Velez, Y. Liao, H. Wang, M. Yang, H. Ji, J. Wang, S. Yan, P. Torr, and L. Bai (2026) The landscape of agentic reinforcement learning for llms: A survey. Trans. Mach. Learn. Res. 2026. External Links: Link Cited by: §1. [63] X. Zhang, J. Wang, Z. Cheng, W. Zhuang, Z. Lin, M. Zhang, S. Wang, Y. Cui, C. Wang, J. Peng, et al. (2025) Srpo: a cross-domain implementation of large-scale reinforcement learning on llm. arXiv preprint arXiv:2504.14286. Cited by: Appendix B. [64] X. Zhang, H. Hu, Q. Ye, L. Bai, and H. Zheng (2025) Mer-inspector: assessing model extraction risks from an attack-agnostic perspective. In Proceedings of the ACM on Web Conference 2025, p. 4300–4315. Cited by: Appendix H. [65] X. Zhang, H. Liu, L. Bai, H. Wang, Q. Ye, T. Zhang, and H. Hu (2026) On the adversarial robustness of large vision-language models under visual token compression. arXiv preprint arXiv:2601.21531. Cited by: Appendix H. [66] Y. Zhang, W. Yu, and G. Turk (2019) Learning novel policies for tasks. In International Conference on Machine Learning, p. 7483–7492. Cited by: §5. [67] Z. Zhang, Y. Wang, J. Liu, G. Sun, and K. Tang (2024) A two-phase kriging-assisted evolutionary algorithm for expensive constrained multiobjective optimization problems. IEEE Transactions on Systems, Man, and Cybernetics: Systems 54 (8), p. 4579–4591. Cited by: §5. [68] Z. Zhang, S. Wu, X. Wu, and K. C. Tan (2026) Semantics-aware bilevel co-evolution: towards automated multicomponent algorithm design. arXiv preprint arXiv:2606.29953. Cited by: §5. [69] Y. Zhou, A. Zanette, J. Pan, S. Levine, and A. Kumar (2024) Archer: training language model agents via hierarchical multi-turn rl. arXiv preprint arXiv:2402.19446. Cited by: Appendix B. [70] Z. Zhou, W. Fu, B. Zhang, and Y. Wu (2022) Continuously discovering novel strategies via reward-switching policy optimization. arXiv preprint arXiv:2204.02246. Cited by: §5. Appendix A Derivation and Implementation of Trajectory-Level Diversity Credits Pairwise diversity credit. For a sampled group of G trajectory descriptors, define pair(Z)=2G(G−1)∑1≤a<b≤Gd(a,b)D_pair(Z)= 2G(G-1) _1≤ a<b≤ Gd(z_a,z_b) (20) and cipair(Z)=1G−1∑j≠id(i,j).c_i^pair(Z)= 1G-1 _j≠ id(z_i,z_j). (21) Let S=∑a<bd(a,b)S= _a<bd(z_a,z_b) and si=∑j≠id(i,j)s_i= _j≠ id(z_i,z_j). Since pair(Z−i)=2(S−si)(G−1)(G−2),D_pair(Z_-i)= 2(S-s_i)(G-1)(G-2), (22) the exact leave-one-out change satisfies pair(Z)−pair(Z−i)=2G−2[cipair(Z)−pair(Z)].D_pair(Z)-D_pair(Z_-i)= 2G-2 [c_i^pair(Z)-D_pair(Z) ]. (23) Moreover, every unordered pair contributes to two credits, so 1G∑i=1Gcipair(Z)=pair(Z). 1G _i=1^Gc_i^pair(Z)=D_pair(Z). (24) Thus, within a fixed group, the exact leave-one-out change is a positive affine transformation of cipairc_i^pair. The two therefore produce identical values after group-wise standardization. Determinantal diversity credit. We construct Kij=exp(−∥i−j∥22σ)K_ij= (- _i-z_j _2^2σ ) (25) and let A=K+ηIA=K+η I. The implemented credit is cidet(Z)=det(A)det(A−i,−i).c_i^det(Z)= (A) (A_-i,-i). (26) Equivalently, logcidet=logdet(A)−logdet(A−i,−i), c_i^det= (A)- (A_-i,-i), (27) so the determinant ratio is the exponential of the leave-one-out log-determinant change. By the Schur-complement identity, cidet(Z)=Aii−Ai,−iA−i,−i−1A−i,i.c_i^det(Z)=A_i-A_i,-iA_-i,-i^-1A_-i,i. (28) Numerical implementation. The pairwise credit uses Euclidean distance. For the determinantal credit, we use σ=10σ=10 and diagonal jitter η=10−4η=10^-4. Raw determinant-ratio credits are divided by their within-group maximum before standardization; this common positive scaling does not affect the subsequent group Z-score. If the regularized full determinant is at most 10−1010^-10, we use the mean off-diagonal kernel dissimilarity 1−Kij1-K_ij as a numerically robust fallback. If an individual leave-one-out determinant is at most 10−1010^-10 while the full determinant is valid, its raw credit is set to one. Groups with zero credit variance receive zero normalized diversity advantage. Appendix B More Discussion on Related Work Given the recent surge of interest in RL for LLMs and agents, there is a substantial body of relevant work worth discussing. We provide a more detailed discussion of these prior studies here. RL for LLMs and Agents. Recent LLMs have shown rapid progress in complex reasoning, with strong performance demonstrated by frontier reasoning models such as OpenAI-o1 [34], DeepSeek-R1 [15], and Kimi-k1.5 [43]. A representative line of work follows the R1-zero training paradigm introduced by DeepSeek-R1 [15], which combines group-based policy optimization with rule-based rewards for final-answer correctness and formatting. Subsequent studies have improved this paradigm along two main directions: improving data quality, curriculum construction, and rollout filtering [26, 33, 17, 57, 21, 18], and refining RL algorithms for more stable and efficient post-training. The latter includes PPO-style and value-based methods for improved advantage estimation and long-chain credit assignment [61, 22, 59, 58], GRPO-style methods that address instability, biased objectives, length or difficulty biases, and sample efficiency [57, 31, 63, 55, 52], as well as REINFORCE-like methods that improve stability or scalability [43, 20, 5]. Relatedly, recent work has explored increasingly automated and modular generation pipelines across long-form video synthesis, agent-driven model merging, and scientific illustration generation [27, 28, 29]. Beyond static reasoning tasks, RL has also been increasingly used to train LLM agents that interact with external environments over multiple turns, including web navigation, application control, embodied tasks, multi-hop information seeking, and other long-horizon interactive settings [47, 46]. Some methods improve agent learning through hierarchical training or guided search, often targeting web-based tasks such as WebShop [69, 38]. Another line studies online RL for interactive agents in device-control or vision-language environments, where exploration in large action spaces is a central challenge [1, 12]. More recent work adapts group-based or trajectory-level RL to long-horizon agent training, focusing on full-trajectory optimization, scalable multi-turn learning, or finer-grained credit assignment [3, 48, 13]. These works primarily target training stability, exploration efficiency, and credit assignment in interactive environments. Appendix C Additional Experimental Details C.1 Environment Details ALFWorld. ALFWorld [41] is a text-based interactive environment aligned with embodied household tasks. Each episode provides a natural-language goal and textual observations of the current environment. At each step, the agent outputs a textual action, such as navigation, object manipulation, or environment inspection. An episode is considered successful if the agent completes the specified household task within the step budget. In our experiments, a trajectory consists of the full sequence of observations, reasoning tokens, actions, and environment feedback: τ=(o1,a1,o2,a2,…,oT,aT).τ=(o_1,a_1,o_2,a_2,…,o_T,a_T). (29) The task reward is defined as rtask=[task success].r^task=1[task success]. (30) Sokoban. Sokoban [39] is a grid-based box-pushing puzzle game. The agent moves on a grid and must push boxes onto target cells. The agent can push boxes but cannot pull them, making the environment sensitive to irreversible action sequences and long-horizon planning. An episode is successful (and receive positive reward) if all boxes are placed on target cells within the step budget. A trajectory consists of the sequence of grid states and actions: τ=(s1,a1,s2,a2,…,sT,aT).τ=(s_1,a_1,s_2,a_2,…,s_T,a_T). (31) We use Sokoban to evaluate whether explicit diversity can capture planning-level behavioral variation rather than tool-use variation. C.2 Training Implementation We implement our method on top of GiGPO [13] using the verl training framework [40]. For each input instance, we sample a group of G trajectories from the current policy. We compute the task reward ritaskr_i^task and descriptor vector iz_i for each trajectory. Given the descriptor matrix Z, we compute the diversity credit ci(Z)c_i(Z) and standardize it within trajectories sampled for the same input: c^i=ci−c¯maxstd(c),10−6. c_i= c_i- c \std(c),10^-6\. (32) For both ALFWorld and Sokoban, the standardized credit is added to the GiGPO advantage: A~i,t=Ai,tGiGPO+λc^i. A_i,t=A_i,t^GiGPO+λ c_i. (33) The trajectory-level signal is broadcast over the valid response tokens associated with trajectory i. The environment-level setup is summarized in Table 6, and the shared optimization and evaluation settings are reported in Table 7. Table 6: Environment-specific experimental setup. Setting ALFWorld Sokoban Base model Qwen2.5-7B-Instruct Qwen3-VL-4B-Instruct Input modality Text Vision-language Environment AlfredTWEnv Sokoban Environment observation Text RGB image Max environment steps 50 15 Training data size 16 32 Total epochs 150 200 Advantage normalization mode mean_std_norm mean_norm Max prompt length 2048 1024 PPO mini-batch size 256 64 PPO micro-batch size per GPU 16 8 Validation frequency (steps) 5 10 Table 7: Shared training and evaluation hyperparameters. Hyperparameter Value Training framework verl RL backbone GiGPO Group size G 8 Max response length 512 Learning rate 1×10−61× 10^-6 KL loss type low-variance KL KL loss coefficient 0.01 Discount factor γ 0.95 GiGPO step-advantage weight ω 1.0 Diversity integration mode advantage Diversity-credit normalization group mean/std Pairwise distance Euclidean DPP kernel bandwidth σ 10 DPP diagonal jitter η 10−410^-4 Diversity-credit standard-deviation floor 10−610^-6 Invalid action penalty coefficient 0.1 Validation temperature 0.4 Standard evaluation presence penalty 0.0 Frequency penalty 0.0 Standard repetition penalty 1.0 Stochastic validation decoding yes Evaluation trajectories per input NevalN_eval 8 Number of GPUs 8 Number of nodes 1 Tensor parallel size 1 GPU memory utilization 0.6 C.3 Reproducibility Runs and randomness. Each reported model configuration is obtained from one training run. The training scripts set the environment seed to 00 for both ALFWorld and Sokoban. Evaluation uses stochastic decoding with temperature 0.40.4. For the main ALFWorld comparison, every checkpoint is evaluated on the same 128 prompts with eight rollouts per prompt, yielding 1,024 trajectories per checkpoint. The GiGPO+P and GiGPO+RP baselines reuse the selected GiGPO checkpoint and change only the vLLM presence penalty or repetition penalty, respectively. They receive no additional training and are not used for checkpoint selection. Diversity metrics are computed within prompt groups over successful trajectories. Compute and software. All reported training runs use one node with eight GPUs and tensor parallel size one. The experiments use eight NVIDIA H20 GPUs with 144 GB memory per GPU. Training is implemented in the PyTorch-based verl framework, with Ray for environment workers and vLLM for rollout generation; the environment uses vLLM 0.17, and pins Transformers 4.51.1. Appendix D Trajectory Descriptor Details This section describes the task-specific trajectory descriptors used in our experiments. Each descriptor maps a trajectory τ to a vector (τ)∈ℝmz(τ) ^m. All descriptor values are computed after trajectory generation and are used as policy-optimization signals during RL training. D.1 ALFWorld Descriptors Action-type descriptor. The action-type descriptor captures the distribution of high-level action categories used by the agent across a trajectory. We parse each environment action into one of m=15m=15 canonical ALFWorld action types: navigation (go to), container manipulation (open, close), object manipulation (pick up, put, take, slice), state-changing operations (clean, heat, cool, toggle, use), and inspection (examine, look, inventory). Action-type matching is performed by string-prefix lookup on the parsed action text. Let type=c1,…,cmA_type=\c_1,…,c_m\ denote the action-category set. For trajectory τ=a1,…,aTτ=\a_1,…,a_T\, the descriptor is the normalized count vector ϕkaction(τ)=1T∑t=1T[type(at)=ck],φ^action_k(τ)\;=\; 1T _t=1^T1\! [type(a_t)=c_k ], (34) for k=1,…,mk=1,…,m, yielding ϕaction(τ)∈Δm−1⊂ℝ15φ^action(τ)∈ ^m-1 ^15. This descriptor encourages diversity in how the agent structures its interaction process, e.g., whether it relies more on navigation, inspection, or manipulation, and is invariant to the specific objects or locations involved. Location descriptor. The location descriptor captures the distribution of navigation destinations visited by the agent during a trajectory. For an ALFWorld input x, let ℒ(x)=ℓ1,…,ℓnL(x)=\ _1,…, _n\ denote the locations available in the corresponding environment. We parse each navigation action of the form go to ℓk _k and define the normalized location-frequency vector ϕklocation(τ)=1Z(τ)∑t=1T[type(at)=go tolocation(at)=ℓk],φ^location_k(τ)\;=\; 1Z(τ) _t=1^T1\! [ subarrayctype(a_t)=go\,to\\ location(a_t)= _k subarray ], (35) for k=1,…,nk=1,…,n, where Z(τ)Z(τ) is the total number of parsed navigation actions. This yields ϕlocation(τ)∈Δn−1⊂ℝ|ℒ(x)|φ^location(τ)∈ ^n-1 ^|L(x)|. The descriptor encourages diversity in route choice and location visitation patterns while remaining invariant to non-navigation action composition. D.2 Sokoban Descriptors For Sokoban we instantiate two complementary trajectory-level behavior descriptors that operate on the agent’s emitted move sequence: a direction-transition descriptor that summarizes transitions between consecutive move directions, and a higher-dimensional spatial-coverage descriptor that captures where on the grid the agent traverses. Both descriptors are extracted from the action stream by parsing action tags inside the agent’s response and are ℓ1 _1-normalized to lie on the probability simplex. Direction-transition descriptor. The direction-transition descriptor captures the distribution of consecutive low-level move-direction pairs used by the agent. Let =d1,d2,d3,d4=up,down,left,rightD=\d_1,d_2,d_3,d_4\=\up,down,left,right\ denote the four cardinal moves, and let T be the number of valid move actions in trajectory τ. For each ordered pair (dp,dq)∈×(d_p,d_q) ×D, we define ϕp,qdir(τ)=1T−1∑t=1T−1[dir(at)=dpdir(at+1)=dq],φ^dir_p,q(τ)\;=\; 1T-1 _t=1^T-11\! [ subarraycdir(a_t)=d_p\\ dir(a_t+1)=d_q subarray ], (36) yielding a normalized transition histogram ϕdir(τ)∈Δ15⊂ℝ16φ^dir(τ)∈ ^15 ^16. This descriptor is invariant to the absolute position on the board and encourages diversity in movement patterns, including straight runs, reversals, and changes between horizontal and vertical motion. Spatial-coverage descriptor. The spatial-coverage descriptor captures where the agent moves on the grid by simulating its trajectory in a board-local coordinate frame. We instantiate a H×WH×W relative grid (we use H=W=7H=W=7 in our experiments) centered at the agent’s start cell, and roll out the action sequence with a clipping policy that keeps the simulated agent inside the grid. Let =g1,…,gmG=\g_1,…,g_m\ with m=HWm=HW enumerate the grid cells, and let ct∈c_t be the simulated cell at step t (with c0c_0 the start cell). The descriptor is the normalized visitation count ϕkcov(τ)=1Z(τ)∑t=0T[ct=gk],k=1,…,m,φ^cov_k(τ)\;=\; 1Z(τ) _t=0^T1\! [c_t=g_k ], k=1,…,m, (37) where Z(τ)=T+1Z(τ)=T+1 normalizes the histogram to the simplex, ϕcov(τ)∈Δm−1⊂ℝ49φ^cov(τ)∈ ^m-1 ^49. Because the simulator anchors the origin at the agent’s start cell rather than the absolute board frame, two trajectories on the same puzzle that visit the same physical cells yield identical descriptors, while trajectories taking distinct routes are pulled apart in ϕcovφ^cov-space. This descriptor encourages diversity in spatial exploration and route choice, and is complementary to ϕdirφ^dir: two policies can share a similar direction distribution yet differ sharply in which grid cells they cover. D.3 Descriptor Summary Table 8: Summary of trajectory descriptors used in our experiments. Environment Descriptor What it captures Vector type ALFWorld action_type High-level action normalized histogram (ℝ15R^15) ALFWorld location Navigation destination usage normalized histogram (ℝ|ℒ(x)|R^|L(x)|) Sokoban direction_transition Consecutive direction pairs normalized histogram (ℝ16R^16) Sokoban spatial_coverage Visited grid-cell distribution normalized 2D occupancy (ℝ7×7R^7×7) Table 8 summarizes how the four descriptors capture complementary behavioral variation in action composition, location visitation, direction transitions, and spatial coverage. Appendix E Determinantal Diversity Evaluation metric. For the determinantal comparison, we report a size-normalized evaluation score derived from the log-determinant functional. Let ZsuccZ_succ contain the descriptors of the n successful trajectories in an evaluation group, and let KsuccK_succ be their RBF Gram matrix. For n≥2n≥ 2, we define Ddpp(Zsucc)=exp(1nlogdet(Ksucc+ηI))=det(Ksucc+ηI)1/n,D^dpp(Z_succ)= \! ( 1n (K_succ+η I) )= (K_succ+η I)^1/n, (38) where Kij=exp(−∥zi−zj∥22/10)K_ij= (- z_i-z_j _2^2/10) and η=10−4η=10^-4. Here, n is the number of successful trajectories, not the descriptor dimension. Groups with fewer than two successes are assigned zero. The transformation is applied before aggregation and is used only for evaluation; it does not change the training-credit definition. Appendix F Location-Diversity Case Study Alternative Navigation Routes. Figure 3: Location-descriptor view of the main-paper route case study for the prompt “put a spraybottle in cabinet.” Each row is one successful rollout, each column is a location activated by at least one rollout, and color denotes the normalized frequency of navigation actions targeting that location. Rows are ordered by their projection onto the first principal component of the pooled GiGPO and TJPO descriptor vectors, and both panels use a shared color scale. Figure 3 provides the descriptor-level view of the same eight-rollout comparison visualized as route graphs in the main paper. The GiGPO rows are nearly identical: seven rollouts allocate equal navigation frequency to Countertop, Dresser, and Cabinet, reflecting the dominant Countertop–Dresser–Cabinet route. In contrast, the TJPO descriptors separate into multiple patterns, including rollouts that navigate directly through the Toilet and rollouts that retain the Dresser alternative. This greater between-rollout heterogeneity raises DlocpairD_loc^pair from 0.1180.118 to 0.3180.318. Together, the route graph and heatmap show that the increase comes from distinct successful navigation strategies rather than a change in task success; both methods solve all eight rollouts. Figure 4: Complete successful environment-action trajectories for the same “put a spraybottle in cabinet” case. All eight sampled rollouts are shown for each method; no trajectories are omitted or selected. Rollouts are grouped by navigation strategy while retaining their original rollout identifiers. Object instance indices are omitted for readability, but every environment action is shown. Colors distinguish navigation destinations and task-completion operations. Figure 4 exposes where the trajectory diversity arises. For GiGPO, seven of eight rollouts execute the same six-action sequence through the Countertop and Dresser; the remaining rollout replaces the Dresser with the Toilet but otherwise follows the same plan, yielding two unique route signatures. TJPO instead produces three successful strategy groups: three five-action rollouts navigate directly to the Toilet, four six-action rollouts inspect the Countertop before using the Toilet, and one rollout retains the Dresser route. The variation is localized to the search and navigation prefix. After a spray bottle is found, all rollouts use the same task-completion structure: take the bottle, navigate to the Cabinet, open it, and place the bottle inside. Thus, TJPO changes which successful route is selected without disrupting the reliable manipulation sequence. The three direct-Toilet trajectories are also one action shorter than the GiGPO trajectories, and none of the TJPO rollouts revisits a location. All 16 displayed trajectories contain zero invalid actions and zero location revisits. The increase in DlocpairD_loc^pair therefore reflects alternative efficient routes rather than longer or repetitive exploration. Alternative Retrieval Locations before Cleaning. Figure 5 provides an additional location-diversity example for “clean some ladle and put it in diningtable.” GiGPO follows the same six-action sequence in all eight rollouts: retrieve a ladle from the DiningTable, clean it at the SinkBasin, and return it to the DiningTable. TJPO follows this branch in six rollouts, while two rollouts retrieve a different ladle from a Countertop before completing the same cleaning and placement operations. Both methods succeed in all eight rollouts. The location-diversity score increases from 00 to 0.2810.281, while the mean action count changes from 6.006.00 to 6.256.25. This example illustrates variation in retrieval locations while preserving the required processing and final placement operations. Figure 5: Complete successful trajectories for “clean some ladle and put it in diningtable.” The top panel shows GiGPO and the bottom panel shows location-pairwise TJPO with λ=0.3λ=0.3. All eight rollouts are retained for each method, with their original rollout identifiers preserved. Object instance indices and singleton-location indices are omitted for readability, while Countertop 1 and Countertop 2 are distinguished. Rows are grouped by navigation sequence; both panels use the same action-position scale, and no actions are omitted. Appendix G Practical Benefits of Behavioral Diversity Evaluation suite. Table 5 reports results on a structurally verified subset of ALFWorld test games, containing 15 tasks from 14 scenes: nine Pick & Place, four Clean & Place, and two Cool & Place tasks. Each task admits successful solutions through two independent source location categories. We retain both intervention directions, yielding 30 cases for each of the User Constraints and Disruption conditions. Task construction. Tasks are selected using PDDL, task metadata, and native environment replay, without using model scores. The two solution branches must use different target object instances. We exclude tasks with shared target objects, or with a forbidden location sharing its symbolic position with an unrestricted named receptacle. Source categories must not overlap the final goal category or required processing tools. Each branch must avoid the other source category, and the lengths of their verified plans must differ by at most two actions. These are feasible witness lengths, not claimed shortest-path lengths. For each source category X, the intervention covers all initially navigable named locations assigned to that category. We verify successful execution using X, avoiding X in the original environment, avoiding X under both intervention conditions, and completing an alternative plan after one rejected access to X. Every successful replay must confirm target pickup and final placement within 50 steps. Both directions must pass these checks; all admitted tasks and directions are retained in the evaluation. Evaluation conditions. Standard uses the original task without additional instructions or changes to the environment. The User Constraints condition adds an instruction to avoid locations of type X, explicitly lists the corresponding locations, and asks the agent to complete the original task. The instruction is repeated in each observation. This condition does not modify admissible actions or environment dynamics, so the agent can violate the requirement. The Disruption condition makes the same predefined location set unavailable to both models from the start of the episode, without initially informing the agent. An attempted navigation or interaction involving this set is rejected and replaced by look, preserving the manipulation state while consuming one interaction step. The first rejection reveals the disruption. Subsequent observations and admissible actions reflect the persistent unavailability of these locations. Neither intervention supplies an alternative location or action plan. Models and sampling. We compare GiGPO with location-pairwise TJPO and λ=0.5λ=0.5. No additional training is performed for these interventions. Each case receives eight stochastic rollouts at temperature 0.40.4 with a maximum of 50 interaction steps. The environment seed is fixed to zero. For each model, the evaluation therefore contains 120 Standard, 240 User Constraints, and 240 Disruption trajectories. Metrics and aggregation. Task success is determined by the native environment. Standard Success is the proportion of successful episodes without intervention. User Constraints Success requires both task completion and no environment-confirmed visit to the forbidden location set. Disruption Success is the proportion of successful episodes under location unavailability, including episodes that never attempt to access a forbidden location. Appendix H Broader Impact This work studies how to explicitly define and optimize trajectory diversity in RL-based post-training for LLM agents. A potential positive impact is improved controllability and robustness: by encouraging multiple high-quality behavioral modes, agents may become less dependent on a single brittle solution path and better adapt to changes in tools, interfaces, or environments. The use of user-specified, task-specific descriptors also makes the diversity objective more interpretable, allowing practitioners to specify which forms of behavioral variation are desirable for a given application. Robustness is also an important concern for multimodal models, where architectural or efficiency-oriented interventions can interact nontrivially with adversarial behavior [65, 64]. At the same time, explicit diversity should be used together with appropriate safety constraints and task validation. In applications involving external tools or real-world operations, practitioners should ensure that diversified behaviors remain safe, reliable, and aligned with intended use.