Paper deep dive
ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning
Xiaoxuan Wang, Han Zhang, Haixin Wang, Yidan Shi, Ruoyan Li, Kaiqiao Han, Chenyi Tong, Haoran Deng, Renliang Sun, Alexander Taylor, Yanqiao Zhu, Jason Cong, Yizhou Sun, Wei Wang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 7/20/2026, 12:40:35 PM
Summary
The paper introduces ARLArena, a framework for stable agentic reinforcement learning (ARL), and proposes SAMPO, a stable policy optimization method. It decomposes policy gradients into four dimensions (loss aggregation, IS clipping, trajectory filtering, advantage design) to identify sources of instability, finding that sequence-level clipping and dynamic sampling improve stability over tolerant clipping.
Entities (22)
Relation Signals (17)
ARLArena → evaluates → AlfWorld
confidence 95% · We adapt ALFWorld... as the agentic tasks.
ARLArena → evaluates → TIR Math
confidence 95% · We adapt ALFWorld, WebShop, Sokoban, and TIR Math as the agentic tasks.
ARLArena → evaluates → WebShop
confidence 95% · We adapt ALFWorld, WebShop, Sokoban, and TIR Math as the agentic tasks.
ARLArena → evaluates → Sokoban
confidence 95% · We adapt ALFWorld, WebShop, Sokoban, and TIR Math as the agentic tasks.
ARLArena → proposes → SAMPO
confidence 95% · Through this fine-grained analysis, we distill a unified perspective on ARL and propose SAMPO
SAPO → causes → Training Collapse
confidence 92% · CISPO and SAPO with tolerant clipping... leads to training instability, with collapse occurring around step 130.
CISPO → causes → Training Collapse
confidence 92% · CISPO and SAPO with tolerant clipping... leads to training instability, with collapse occurring around step 130.
GSPO → outperforms → GRPO
confidence 90% · GSPO consistently outperforms all other policy optimization methods, achieving an average improvement of 13.3% compared to GRPO.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Agentic reinforcement learning (ARL) has rapidly gained attention as a promising paradigm for training agents to solve complex, multi-step interactive tasks. Despite encouraging early results, ARL remains highly unstable, often leading to training collapse. This instability limits scalability to larger environments and longer interaction horizons, and constrains systematic exploration of algorithmic design choices. In this paper, we first propose ARLArena, a stable training recipe and systematic analysis framework that examines training stability in a controlled and reproducible setting. ARLArena first constructs a clean and standardized testbed. Then, we decompose policy gradient into four core design dimensions and assess the performance and stability of each dimension. Through this fine-grained analysis, we distill a unified perspective on ARL and propose SAMPO, a stable agentic policy optimization method designed to mitigate the dominant sources of instability in ARL. Empirically, SAMPO achieves consistently stable training and strong performance across diverse agentic tasks. Overall, this study provides a unifying policy gradient perspective for ARL and offers practical guidance for building stable and reproducible LLM-based agent training pipelines.
Tags
Links
- Source: https://arxiv.org/abs/2602.21534v3
- Canonical: https://arxiv.org/abs/2602.21534v3
Trouble viewing inline? Open PDF directly →
Full Text
94,296 characters extracted from source content.
Expand or collapse full text
ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning Xiaoxuan Wang * 1 Han Zhang * 1 Haixin Wang * 1 Yidan Shi † 1 Ruoyan Li † 1 Kaiqiao Han † 1 Chenyi Tong 2 Haoran Deng 1 Alexander K Taylor 1 Renliang Sun 1 Yanqiao Zhu 1 Jason Cong 1 Yizhou Sun 1 Wei Wang 1 Abstract Agentic reinforcement learning (ARL) has rapidly gained attention as a promising paradigm for training agents to solve complex, multi-step interactive tasks. Despite encouraging early re- sults, ARL remains highly unstable, often lead- ing to training collapse. This instability lim- its scalability to larger environments and longer interaction horizons, and constrains systematic exploration of algorithmic design choices. In this paper, we first propose ARLArena, a sta- ble training recipe and systematic analysis frame- work that examines training stability in a con- trolled and reproducible setting.ARLArena first constructs a clean and standardized testbed. Then, we decompose policy gradient into four core design dimensions and assess the perfor- mance and stability of each dimension. Through this fine-grained analysis, we propose SAMPO, a stable agentic policy optimization method de- signed to mitigate the dominant sources of in- stability in ARL. Empirically, SAMPO achieves consistently stable training and strong perfor- mance across diverse agentic tasks. Our code is open-sourced at: https://github.com/ WillDreamer/ARL-Arena.git 1. Introduction Large language models (LLMs) have been increasingly de- ployed as autonomous agents for complex, multi-step in- teractive tasks spanning web navigation (Zhou et al., 2024; Wang et al., 2026), embodied environments (Shridhar et al., 2020), games (Xi et al., 2024), and deep research (Jin et al., 2025; Guan et al., 2025). These tasks demand planning, * Equal contribution † Contributed equally as second authors 1 Department of Computer Science, University of California, Los Angeles, California, USA 2 University of Wisconsin–Madison, Madison, Wisconsin, USA. Correspondence to: Xiaoxuan Wang <xw27@g.ucla.edu>, Haixin Wang <whx@g.ucla.edu>. Proceedings of the43 rd International Conference on Machine Learning, Seoul, South Korea. PMLR 306, 2026. Copyright 2026 by the author(s). tool use, and long-horizon decision-making, necessitating training objectives that capture multi-turn interactions. Re- inforcement learning (RL) offers a principled post-training framework for this purpose, building on its success in static reasoning tasks (e.g., DeepSeek-R1 (Guo et al., 2025), OpenAI o1 (Jaech et al., 2024)), and early results in the agentic setting are promising (Jin et al., 2025; Cheng et al., 2025; Xi et al., 2024). However, agentic RL (ARL) training remains highly unsta- ble and prone to collapse (Xi et al., 2025). This instability arises from the interactive, multi-turn nature of agentic en- vironments, which introduce compounding challenges such as invalid actions, sparse rewards, long-horizon credit as- signment, and non-stationary agent–environment dynam- ics (Wang et al., 2025b; Xu et al., 2026). Small deviations in early decisions can cascade across turns, causing distri- bution shifts that amplify credit-assignment noise and pro- duce degenerate rollouts (Xia et al., 2026; Xie et al., 2026). Consequently, ARL outcomes are difficult to reproduce across runs and environments, and scaling to longer hori- zons or more complex interaction spaces remains severely limited (Abdulhai et al., 2023; Xi et al., 2025). These chal- lenges underscore the need for stable and scalable training solutions for ARL. This paper addresses this gap by introducing ARLArena, a stable training recipe and systematic analysis framework for agentic reinforcement learning. We first construct a clean, standardized testbed through format correction, be- havior cloning initialization, and KL-based regularization, establishing reliable baseline performance. We then de- compose policy-gradient–based RL into four orthogonal design dimensions and evaluate the effectiveness and sta- bility of each across diverse agentic tasks. Each dimension is examined in isolation using representative policy opti- mization (PO) methods; for methods that exhibit training collapse, we further diagnose the underlying failure modes and develop targeted stabilization strategies. This systematic analysis yields three key findings: (1) tol- erant clipping induces training collapse, whereas sequence- level clipping ensures stable improvement; (2) incorporat- ing environment-level information into advantage design improves both stability and performance; and (3) dynamic 1 arXiv:2602.21534v3 [cs.AI] 4 Jul 2026 ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning 050100150 Training Steps 0.0 0.2 0.4 0.6 0.8 1.0 Success Rate ALFWorld 050100150 Training Steps 0.0 0.2 0.4 0.6 0.8 1.0 Sokoban DAPOSAPOGSPOEMPGGIGPOGRPOCISPOSAMPO (Ours) Figure 1. Training curves on ALFWorld (left) and Sokoban (right). Our proposed SAMPO achieves the highest success rates on both environments with stable, monotonic improvement throughout training, while baseline methods exhibit varying degrees of instability. These results demonstrate that principled integration of sequence-level clipping, advantage design, and dynamic filtering, as combined in SAMPO, is critical for both training stability and final performance in multi-turn ARL. sampling combined with fine-grained advantage design fur- ther benefits ARL training. Motivated by these insights, we propose S table Agentic Multi-turn Policy Optimization (SAMPO), a unified PO method that directly addresses the dominant sources of instability identified in our anal- ysis. SAMPO consistently improves training stability and performance, achieving an average 25.2% improvement over the GRPO baseline. We additionally study the impact of off-policy staleness in agentic environments and con- duct comparative evaluations against proprietary models, demonstrating the robustness and generality of ours. In summary, our contributions are: (i) a unifying policy gradient perspective and four-dimensional categorization of PO methods for ARL; (i) a standardized, reproducible testbed and diagnostic methodology for multi-turn ARL stability; (i) principled, task-robust findings and remedies for common collapse modes; and (iv) SAMPO, a new PO method that achieves both reliable training and strong final performance. We hope this study provides a foundation for more principled progress in agentic post-training. 2. Problem Formulation 2.1. Policy Gradient for Agentic RL During RL optimization for LLMs, the policy π θ generates a response trajectory y = (y 0 ,...,y T ) conditioned on a prompt x, which is subsequently used for policy updates (Ouyang et al., 2022). Following PPO-style optimization (Schulman et al., 2017), trajectories collected under a be- havior policy π θ old are used to update the current policy π θ . The corresponding policy gradient can be written as: ∇ θ L(θ) =E y∼π θ old " T X t=0 w t (y)∇ θ logπ θ (y t | x,y <t ) A(x,y) # , (1) where the importance sampling weight is given by: w t (y) = P θ (y t | x,y <t ) P θ old (y t | x,y <t ) = π θ (y t | x,y <t ) π θ old (y t | x,y <t ) .(2) Here, A(x,y) represents the advantage. Agentic RL. An agent interacts with the environment over K turns, forming a long-horizon decision-making pro- cess (Wei et al., 2026; Luo et al., 2026). At each turn, the policy conditions on the accumulated history to generate a response, from which an action is extracted and executed to transition the environment state. The initial user prompt is x (1) . At turn k ∈1,...,K, the policy generates a response y (k) ∼ π θ (· | x (k) ). Given the environment state s (k) , actions a (k) are extracted from y (k) , and the environment transitions to the next state s (k+1) ac- cording to an update function f : s (k+1) = f a (k) ,s (k) , where f(·) is the state transition function that incorporates tool calls, environment observations, or retrieved informa- tion. The user prompt for turn k + 1, denoted x (k+1) , is constructed from the updated state s (k+1) . Finally, the complete multi-turn interaction trajectory is defined as τ = x (1) ,y (1) ,x (2) ,y (2) ,...,x (K) ,y (K) . In the multi-turn agent–environment setting described above, we decompose a K-turn trajectory into single-turn updates. This yields the following policy gradient formula- tion for agentic LLM interaction: ∇ θ L(θ) =E τ∼π θ old h K X k=1 T k X t=0 w t (y (k) ) | z IS ∇ θ logπ θ y (k) t | x (k) ,y (k) <t | z Log prob A(x (k) ,y (k) ) | z Advantage i . (3) 2.2. Policy Gradient Decomposition Dimensions According to Equation 3, the policy gradient formulation for agentic LLMs can be decomposed into four key re- 2 ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning MethodLoss ObjectiveAdvantage (A i ) IS (w t ) Clipping Dynamical Sampling Adv < 0Adv > 0 GRPO 1 P G i=1 T i G X i=1 T i −1 X t=0 min w t A i , clip(w t , 1±ε)A i r i − mean(r i ) std(r i ) 1− ε, w t < 1− ε, w t ,otherwise. 1 + ε w t > 1 + ε, w t ,otherwise. × GRPO ST 1 G G X i=1 1 T i T i −1 X t=0 min w t A i , clip(w t , 1±ε)A i r i − mean(r i ) std(r i ) 1− ε, w t < 1− ε, w t ,otherwise. 1 + ε, w t > 1 + ε, w t ,otherwise. × GRPO SM 1 P G i=1 T i G X i=1 T i −1 X t=0 M i min w t A i , clip(w t , 1±ε)A i M i = 1 A i ≥ 0 or 1 |T i | |T i |−1 X t=0 log π θ old (y t |x,y <t ) π θ (y t |x,y <t ) ≤ δ r i − mean(r i ) std(r i ) 1− ε, w t < 1− ε, w t ,otherwise. 1 + ε, w t > 1 + ε, w t ,otherwise. × SAPO 1 P G i=1 T i G X i=1 T i −1 X t=0 f i,t (w t )A i r i − mean(r i ) std(r i ) σ(τ neg (w t − 1))· 4 τ neg σ(τ pos (w t − 1))· 4 τ pos × CISPO 1 P G i=1 T i G X i=1 T i −1 X t=0 sg(w t )A i logπ θ r i − mean(r i ) std(r i ) 1− ε low , w t < 1− ε low , sg(w t ),otherwise. 1 + ε high , w t > 1 + ε high , sg(w t ),otherwise. × GSPO 1 P G i=1 T i G X i=1 T i −1 X t=0 min s i A i , clip(s i , 1±ε)A i s i = exp 1 |T i | |T i |−1 X t=0 log π θ (y t | x,y <t ) π θ old (y t | x,y <t ) r i − mean(r i ) std(r i ) 1− ε, s i < 1− ε, s i ,otherwise. 1 + ε, s i > 1 + ε, s i ,otherwise. × GIGPO 1 P G i=1 T i G X i=1 T i −1 X t=0 min w t A ′ i,k , clip(w t , 1±ε)A ′ i,k A i + ω· A step (ˆy i,k ) 1− ε, w t < 1− ε, w t ,otherwise. 1 + ε, w t > 1 + ε, w t ,otherwise. × EMPG 1 P G i=1 T i G X i=1 T i −1 X t=0 min w t A ′ i , clip(w t , 1±ε)A ′ i g H k A i + ζ f H k+1 1− ε, w t < 1− ε, w t ,otherwise. 1 + ε, w t > 1 + ε, w t ,otherwise. × DAPO 1 P G i=1 T i G X i=1 T i −1 X t=0 min w t A i , clip(w t , 1±ε)A i r i − mean(r i ) std(r i ) 1− ε low , w t < 1− ε low , w t ,otherwise. 1 + ε high , w t > 1 + ε high , w t ,otherwise. ✓ Table 1. A summary of policy optimization methods studied in ARLArena, decomposed along four design dimensions: loss objective formulation, advantage (A i ), importance sampling (IS) clipping, and dynamic sampling. Colored entries highlight distinctive design choices: purple denotes modified loss aggregation (seq-mean-token-mean), violet indicates alternative IS clipping strategies (tolerant or sequence-level), and green marks novel advantage designs. The importance sampling weight isw t = π θ (y t | x,y <t )/π θ old (y t | x,y <t ), and sg(·) denotes the stop-gradient operator. search dimensions: Loss Aggregation, Importance Sam- pling (IS) clipping, Trajectory Filtering and Resampling, and Advantage Design. To study each dimension in isola- tion, we analyze the batch-level loss objective without loss of generality. We summarize mainstream PO algorithms across the different design dimensions of the policy gradi- ent in Table 1. Loss Aggregation. In practice, we approximate the loss objective using different loss aggregation schemes. L(θ) =E y (i) ∼π θ old h E t ℓ i,t (θ) i ≜ 1 N N X i=1 1 T i T i −1 X t=0 ℓ i,t (θ) (seq-mean-token-mean)(4) ≜ 1 P N i=1 T i N X i=1 T i −1 X t=0 ℓ i,t (θ) (token-mean),(5) where ℓ i,t (θ) := min w i,t (θ)A i , clip w i,t (θ), 1−ε, 1+ ε A i . N denotes the total number of decomposed turns over trajectories. A i denotes the advantage of sequence y (i) , and w i,t (θ) is the importance sampling ratio at token t of sequence y (i) . Seq-mean-token-mean weights each to- ken by the inverse of its trajectory length, biasing optimiza- tion toward shorter trajectories and potentially introduc- ing response-level length bias. Token-mean assigns equal weight to all unmasked tokens in the batch. Additional ag- gregation strategies are provided in the Appendix A.1. IS Clipping. Clipping methods constrain the magnitude of policy updates by limiting the change in action probabil- ities relative to the old policy. By constraining the deviation between the new and old policies within a bounded range, clipping mitigates performance degradation and instability caused by excessively large policy updates. The loss objec- tive is formulated as follows: L(θ) = 1 P N i=1 T i N X i=1 T i −1 X t=0 min w i,t (θ)A i , clip w i,t (θ), 1±ε A i . (6) Within the GRPO (Guo et al., 2025) framework, several clipping variants are considered, including CISPO (Chen et al., 2025), SAPO (Gao et al., 2025), and GSPO (Zheng et al., 2025). CISPO employs a stop-gradient mechanism to avoid hard clipping of out-of-bounds tokens while pre- serving their gradient information. SAPO adopts a soft- clipping strategy, in which excessively large ratios are smoothly attenuated rather than truncated. GSPO performs clipping by using the sequence-level importance ratio as the clipping criterion. Detailed formulations of these variants 3 ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning AlgorithmStrategyTask ScoreSuccess Rate GRPO + Behavior Cloning+ 2.56+ 20.71 +R format + 0.49+ 7.34 + KL k 3 (x)+ 0.95+ 18.10 GSPO ε : e −2 → e −3 + 0.70+ 3.36 ε : e −3 → e −4 − 1.16 − 9.88 DAPOMaxtry: 2→ 3+ 0.59+ 22.15 SAPO Temperature: 1→ 2 − 1.20− 9.85 Temperature: 2→ 3 − 0.70 − 9.20 Table 2. Incremental stabilization strategies for constructing a standardized testbed on ALFWorld, evaluated using GRPO as the base policy optimizer. Each row adds one stabilization technique or adjusts a method-specific hyperparameter. Task Score and Suc- cess Rate report the absolute improvement (+) or degradation (−) relative to the preceding configuration. are provided in Table 1 and Appendix A.2. Trajectory Filtering and Resampling. Dynamic sam- pling addresses inefficiency caused by zero-gradient trajec- tories in long-horizon agent training (Yu et al., 2025a). L(θ) = 1 P N i=1 T i N X i=1 T i −1 X t=0 min w i,t (θ)A i , clip w i,t (θ), 1±ε A i , s.t. 0 < n y (i) is equivalent(a,y (i) ) o < G. (7) Here, a denotes the ground-truth task completion target, and equivalence is determined by whether the agent suc- cessfully completes the task. It adaptively filters out trajec- tories whose sampled output groups receive identical re- wards (e.g., all correct or all incorrect) and resamples ad- ditional trajectories to increase the proportion of samples with informative gradient signals. Advantage Design. Multi-turn agentic reinforcement learning introduces additional interaction steps and explicit agent–environment state transitions, which motivates spe- cialized advantage designs. GiGPO (Feng et al., 2025) de- fines advantages at the state level by grouping actions con- ditioned on the same preceding environment state and as- signing them a shared relative advantage. EMPG (Wang et al., 2025a) augments the advantage function with an entropy-dependent term, which modulates the learning sig- nal at each turn to better account for uncertainty across in- teraction steps. Detailed formulations of these variants are provided in Table 1 and Appendix A.3. 3. Experimental Setup 3.1. Standardized Testbed A primary challenge is constructing a fair and effective testbed for comparing different algorithms. To address this issue, we progressively apply a sequence of stabilization strategies shown in Table 2. Specifically, we start with be- havior cloning, followed by format penalty enforcement and KL regularization when necessary, and finally PO- specific hyperparameter tuning. This process yields a stan- dardized and stable testbed that provides a solid foundation for systematically comparing different PO strategies. (1) Behavior Cloning. We first perform behavior cloning (BC) on supervised interaction traces to initialize the policy within a reasonable behavioral manifold. Specifically, we construct a multi-turn SFT dataset by deploying the Qwen3 series model (Yang et al., 2025) in the target training en- vironments, collecting self-generated interaction trajecto- ries, and retaining only high-scoring rollouts for supervi- sion. This self-bootstrapped SFT stage initializes the pol- icy within a reasonable behavioral manifold aligned with the environment dynamics. (2) Format Penalty. We incorporateR format that enforces structured outputs with explicit <think> </think> and <action> </action> tags. If the generated out- put violates this format (e.g., missing tags, malformed nesting, or extraneous content outside the tags), we ap- ply a fixed penalty to the final reward.This explicit structural constraint provides dense shaping signals during early training and substantially reduces invalid rollouts that would otherwise corrupt policy updates. (3) Auxiliary KL Loss. Unconstrained updates may cause the policy to drift excessively from the reference model. To regularize policy updates and preserve the pre- trained knowledge embedded in the base model, we intro- duce a KL divergence penalty between the current policy π θ and a reference policy π ref . This constraint encourages conservative policy improvement while still allowing suf- ficient exploration in the action space. We adopt the com- monly used Bregman divergence estimator k 3 for KL ap- proximation, which leverages control variates to achieve unbiasedness and low variance (Schulman, 2017). Specifi- cally, k 3 is defined as k 3 (x) = δ(x)− 1− logδ(x), where δ(x) = p(x) q(x) denotes the likelihood ratio. (4) PO-specific Hyper-parameter Grid Search. A nat- ural question is how to ensure that each PO method is fairly evaluated in the multi-turn setting. Our solution is to first run each method with its default configuration, and then perform a PO-specific hyperparameter grid search. We continue tuning until the training trajectory becomes stable, measured by the variance of the success rate over the final 20% of training steps falling below a predefined threshold. As shown in Table 1, hyperparameters related to IS clip- ping are particularly sensitive. The best-performing con- figurations and full results are reported in Appendix B. 4 ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning 3.2. Tasks and Training Details We adapt ALFWorld (Shridhar et al., 2020), Web- Shop (Yao et al., 2022), Sokoban (Schrader, 2018), and TIR Math (Xue et al., 2025) as the agentic tasks. Our en- tire codebase is built upon the verl RL framework (Sheng et al., 2025).We employ an agentic-loop architecture to coordinate rollouts and environment interactions, after which we segment each complete trajectory into multiple single-turn samples for policy optimization. For mathemat- ical tasks, we use Qwen3-4B-base as the policy model, while for all other tasks we initialize from the SFT-tuned variant Qwen3-4B. For consistency validation, we addi- tionally employed SFT-tuned Qwen3-8B, and the corre- sponding experimental results are provided in Appendix C. All experiments are conducted on NVIDIA H200 or B200 GPUs. Key hyperparameters and training details are re- ported in Appendix B. 4. Exploring Gradient Dimensions on ARL The experimental results for all policy optimization meth- ods are reported in Table 3. GRPO ST denotes GRPO with sequence-mean-token-mean loss aggregation. DAPO GRPO and DAPO GIGPO denote GRPO and GIGPO augmented with dynamic filtering, respectively. 4.1. Impact of IS on ARL We study GSPO, CISPO, and SAPO along the importance- sampling (IS) dimension. GSPO adopts sequence-level clipping, while CISPO and SAPO employ tolerant clip- ping techniques. For CISPO and SAPO, we further ap- ply sequence masking (denoted as CISPO SM and SAPO SM ) to improve training stability. Detailed training dynamics are reported in Figure 2, with IS token-level and sequence- level analyses presented in Figure 3. Table 3 shows that CISPO and SAPO perform substantially worse than GRPO across all tasks, achieving average scores of 34.03 and 32.22, respectively, compared to 46.16 for GRPO. In con- trast, GSPO consistently outperforms all other policy op- timization methods, achieving an average improvement of 13.3% compared to GRPO. To understand training behavior beyond final performance, we analyze training dynamics from multiple perspectives across several metrics. Different IS designs induce vary- ing distances between the current policy and both the be- havior and reference policies during training. These dis- tance variations, in turn, influence optimization behavior (reflected by gradient norms), impact data quality (through the valid action ratio), and ultimately affect task success rates. Jointly examining these metrics enables a more com- prehensive understanding of training stability and failure modes. Figure 2 reports success rate, off-policy KL di- vergence (between the new and old policies), KL loss (be- tween the new and reference policies), gradient norm, and the valid-format ratio of rollout action tokens. As shown in Figure 2, CISPO and SAPO with tolerant clip- ping exhibit rapid initial performance gains, characterized by higher success rates, larger policy updates relative to the reference model, and faster format ratio adaptation com- pared to GRPO and GSPO. This behavior indicates more aggressive optimization that departs quickly from the ref- erence policy and adapts rapidly to the task. A possible explanation is that tolerant clipping may preserve gradient contributions from tokens that deviate substantially from the current policy, resulting in overly exploratory updates. However, such aggressiveness leads to training instability, with collapse occurring around step 130. This collapse is marked by exploding gradient norms and KL divergence, accompanied by a sharp drop in the valid-format ratio, ulti- mately resulting in a severe degradation of success rate. In contrast, GSPO demonstrates a substantially more stable training pattern, with gradual performance improvement accompanied by steady KL divergence and gradient norms. These results indicate that sequence-level clipping is effec- tive for stabilizing training, while overly tolerant clipping thresholds may yield short-term gains at the cost of long- term stability. Furthermore, IS design substantially impacts both performance and training stability in ARL, making it an important dimension in ARL system design. Finding 1 ARL is highly sensitive to IS design: tolerant clip- ping yields fast early gains but causes training col- lapse, whereas sequence-level clipping ensures sta- ble improvement. Rooted cause of training collapse. To investigate the root causes of training collapse along the IS dimension, we analyze token-level importance ratio statistics and stratify sequences by IS ratio, advantage, and entropy for SAPO and SAPO SM , where SAPO SM denotes a stabilized variant of SAPO introduced later. Figure 3 reports token-level and sequence-level IS ratio analysis. Subfigures (a) and (b) present the statistics of tokens whose importance sam- pling ratios fall outside the standard clipping range. Specif- ically, we report the proportion of out-of-bounds tokens and decompose it into lower- and upper-bound portions. The lower-bound portion corresponds to negative-advantage to- kens with importance ratios below ε low , while the upper- bound portion corresponds to positive-advantage tokens with ratios exceeding ε high . As shown in Figure 3, during the collapse stage, SAPO ex- hibits a rapidly growing number of out-of-bounds tokens, predominantly from negative-advantage sequences with 5 ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning DimensionMethod ALFWorldWebShopSokobanTIR Math Avg ScoreSuccessScoreSuccessScoreSuccessAIMEAIME25 BaseGRPO3.7062.3675.3257.715.5183.9049.9630.7846.16 (48.08) Loss AggGRPO ST 4.41↑ 19.2%72.61↑ 16.4%64.57↓ 14.3%51.29↓ 11.1%3.03↓ 45.0%68.73↓ 18.1%27.55↓ 44.9%21.63↓ 29.7%39.23↓ 15.0% Importance Sampling SAPO0.80↓ 78.4%25.16↓ 59.7%73.85↓ 1.9%52.10↓ 9.7% −0.23↓ 104%30.25↓ 63.9%45.00↓ 9.9%30.85↑ 0.2%32.22↓ 30.2% CISPO2.16↓ 41.6%54.42↓ 12.7%67.96↓ 9.8%54.71↓ 5.2% −0.47↓ 109%26.02↓ 69.0%36.53↓ 26.9%30.87↑ 0.3%34.03↓ 26.3% GSPO5.19↑ 40.3%78.61↑ 26.1%85.29↑ 13.3%72.48↑ 25.6%5.22↓ 5.3%82.22↓ 1.7%51.29↑ 2.7%37.95↑ 23.3%52.28↑ 13.3% Advantage Design GIGPO4.97↑ 34.3%81.09↑ 30.0%67.76↓ 10.0%56.55↓ 2.0%5.19↓ 5.8%82.67↓ 1.5%–49.71↑ 3.4% EMPG3.32↓ 10.3%57.91↓ 7.1%79.16↑ 5.1%64.32↑ 11.5%4.48↓ 18.7%79.16↓ 5.6%–48.06↓ 0.1% Dynamic Sampling DAPO GRPO 1.95↓ 47.3%49.58↓ 20.5%62.43↓ 17.1%46.17↓ 20.0%5.16↓ 6.4%82.40↓ 1.8%54.66↑ 9.4%38.97↑ 26.6%42.67↓ 7.6% DAPO GIGPO 2.49↓ 32.7%60.55↓ 2.9%88.10↑ 17.0%76.82↑ 33.1%6.01↑ 9.1%86.20↑ 2.7%–53.36↑ 11.0% OursSAMPO7.04↑ 90.3%92.72↑ 48.7%88.37↑ 17.3%77.73↑ 34.7%6.56↑ 19.1%88.86↑ 5.6%–60.21↑ 25.2% Table 3. Performance comparison of policy optimization methods across four agentic tasks, evaluated on the SFT version of Qwen3- 4B. Methods are organized by their primary design dimension: loss aggregation, importance sampling clipping, advantage design, and dynamic sampling. Green/red subscripts denote the percentage improvement/degradation relative to the GRPO baseline. SAMPO (ours) achieves the highest average score (59.55) with consistent gains across ALFWorld (92.72% success), WebShop (74.08% success), and Sokoban (88.86% success). The evaluation metric for TIR Math is Pass@4; “–” indicates the method is not applicable. For GRPO, the value in parentheses reports the average over the first three tasks only. Figure 2. Training dynamics of six IS variants on ALFWorld: GRPO, GSPO, SAPO, CISPO, and their sequence-masked counterparts SAPO SM and CISPO SM . Panels show (from left to right) success rate, off-policy KL divergence between the current and behavior policies, KL loss between the current and reference policies, gradient norm, and valid-format ratio of rollout actions. small importance ratios (the lower-bound portion). In con- trast, for stable training runs, the portion of out-of-bounds tokens remains fairly low, and lower- and upper-bound ra- tio portions remain relatively balanced. This growing pat- tern and imbalance during collapse suggests that negative- advantage samples with low IS ratios are the main contrib- utors the observed training instability. Beyond token-level analysis, we conduct a sequence-level comparison across training steps in Subfigures (c) and (d). Rollout samples are partitioned according to three factors: the sign of the advantage, whether the importance ratio is smaller or larger than one, and whether policy entropy falls below or exceeds a predefined threshold. This yields eight groups per training step. The vertical area denotes the nor- malized KL divergence between the current policy and the reference policy. A larger area therefore corresponds to a greater deviation from the reference policy, indicating a stronger contribution to policy shift during training. For collapsed experiments, the proportion of KL divergence at- tributed to sequences with negative advantages and low im- portance ratios increases abruptly, whereas for stable train- ing this KL distribution remains relatively balanced across groups. Entropy is less impactful than advantage and IS ratio. This pattern further reinforces the conclusion that negative-advantage samples with low importance ratios are a primary source of training instability. Stabilization Strategies for SAPO and CISPO. We ex- plore several strategies to stabilize SAPO and CISPO train- ing, reported in Table 4. First, we consider increasing the KL coefficient to regularize optimization, and enlarging the mini-update batch size to mitigate off-policy effects. As shown in Table 4, increasing the KL coefficient overly con- strains training and yields limited performance gains (full success-rate plots reported in Appendix C). Similarly, in- creasing the mini-update batch size degrades performance. Motivated by the IS-token analysis during training col- lapse, we adopt sequence masking following (Liu et al., 2025) to directly control negative samples that induce in- stability. Specifically, sequences with negative advantages and low importance ratios are masked (see Table 1 for the detailed formulation), a variant we denote as GRPO SM . We apply sequence masking to SAPO and CISPO, denoted as SAPO SM and CISPO SM . According to Figure 3 and Table 4, applying sequence masking improves the success rate from 6 ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning (a)(b) (c)(d) Figure 3. Token-level and sequence-level IS analysis of SAPO and its sequence-masked variant SAPO SM . (a, b) Fraction of tokens with importance ratios outside the clipping range, decomposed into lower-bound (negative advantage) and upper-bound (positive advantage) portions. (c, d) Rollout groups partitioned by advantage sign, entropy level, and IS ratio magnitude, with KL divergence normalized for relative comparison. MethodMetricOriginalKL (0.05)Off-Policy (1024)Seq-Mask CISPO Score2.161.600.985.25 Success54.4238.4621.5978.88 SAPO Score0.802.403.824.88 Success25.1648.0564.3076.92 Table 4. Effect of different stabilization strategies on CISPO and SAPO in ALFWorld. We evaluate three stabilization techniques applied to the tolerant-clipping methods CISPO and SAPO: in- creasing the KL penalty coefficient to 0.05, enlarging the off- policy mini-update batch size to 1024, and applying sequence- level masking (Seq-Mask). 54.12 to 78.88 for CISPO and from 25.16 to 76.92 for SAPO. SAPO SM and CISPO SM effectively stabilizes train- ing, yielding success rates comparable to GSPO, along with steady KL divergence and gradient norms (Figures 2, 3). Finding 2 Training collapse is largely driven by the accumu- lation of negative-advantage sequences with low IS ratios. Sequence masking of such sequences stabi- lizes training. 4.2. Impact of Advantage Design on ARL We study GIGPO and EMPG along the advantage-design dimension. GIGPO incorporates both global and local ad- vantage information from the environment, enabling fine- grained advantage estimation, while EMPG reshapes ad- vantages by incorporating uncertainty information from the training data. Table 3 shows that GIGPO generally outperforms GRPO, achieving an average score of 49.71 compared to 48.08, with a particularly strong improvement of 34.4% on ALF- World. In addition, EMPG exhibits task-dependent perfor- mance, improving the success rate on WebShop by 11.5% while degrading performance on ALFWorld by 7.1%, re- sulting in an average score difference of 0.1 compared to GRPO. This suggests that fine-grained advantage de- sign incorporating richer environmental information im- proves performance and alleviates reward sparsity in ARL, whereas advantage reshaping based on uncertainty signals has a smaller effect. Finding 3 Incorporating fine-grained environmental advan- tage in ARL improves performance. 4.3. Impact of Dynamic Filtering on ARL Dynamic filtering is well known for delivering strong performance improvements on mathematical reasoning tasks (Yu et al., 2025a; Xue et al., 2025). However, we find that these gains do not always transfer to ARL set- tings. As shown in Table 3, dynamic filtering improves per- formance more consistently when combined with GIGPO than with GRPO. This difference stems from how dy- namic filtering interacts with format learning. In early training, many rollout groups fail entirely due to format errors, which amplifies the format penalty and produces strong implicit advantage signals for format correction. As a result, the model rapidly acquires correct format- ting from early rollouts. Meanwhile, dynamic filtering re- moves such all-failure groups. For GRPO, whose advan- 7 ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning ALFWorldMath Degree ScoreSuccess AIMEAIME25 k@1k@32k@1k@32 Low3.5060.8026.9587.3424.6150.00 Medium3.8358.3824.2275.0017.9748.59 High2.3352.7119.5374.9916.4143.85 Table 5. Effect of off-policy staleness on ALFWorld and MATH. We vary the degree of off-policy staleness (Low, Medium, High) and report task score, success rate (ALFWorld), and pass@k ac- curacy (AIME, AIME25). tage signals have limited diversity, filtering substantially reduces format-related learning signals, leading to unsta- ble format behavior and limited gains. In contrast, GIGPO produces more diverse advantage signals, which stabilize format learning even after filtering, allowing DAPO GIGPO to achieve better and more stable performance. The de- tailed evidence is provided in Appendix G. Finding 4 Dynamic filtering with GIGPO is beneficial for training stability and performance in ARL. 4.4. Impact of Loss Aggregation on ARL As shown in Table 3, sequence-mean-token-mean loss ag- gregation (GRPO ST ) degrades performance from 46.16 to 39.23 relative to token-mean aggregation (GRPO). Al- though GRPO ST yields a 16.4% improvement on ALF- World, it leads to a substantial decline on TIR-Math, with a 44.9% decrease on AIME. Notably, math rollouts exhibit higher variance in sequence length compared to other tasks, ranging from brief solutions to extended reasoning traces. These findings suggest that the unbalanced token weight- ing induced by sequence-level aggregation may negatively affect ARL training, particularly in tasks characterized by high length variability. 4.5. Further Stability Considerations Exploration on Off-Policy Staleness. Due to infrastruc- ture and efficiency constraints, policy training is typically performed in batched rollouts, where groups of trajectories are generated and updated sequentially before proceeding to the next rollout stage. Off-policy effects arise because later updates within the same rollout stage use data from an earlier policy while the current policy has already evolved. Such off-policy mismatch is further amplified in multi- turn settings, where turn-wise decomposition increases the number of samples subject to staleness. Experiment Setup and Results. We control off-policy degree through rollout configuration while holding the up- date batch size fixed. For TIR Math, rollout batch sizes of 128, 512, and 1024 correspond to low, medium, and high off-policy degrees, respectively. For ALFWorld, we vary the off-policy degree by adjusting the number of groups per rollout to 8, 16, and 32. The effects of off-policy stale- ness are summarized in Table 5. TIR Math achieves higher performance under a low off-policy ratio (rollout batch size = 128), with 87.34% and 50.00% for pass@32, compared to 74.99% and 43.85% under a high off-policy ratio. Simi- larly, ALFWorld attains its highest success rate of 60.80% under low off-policy settings, which decreases to 52.71% under high off-policy settings. These results suggest that policy gradient optimization for agentic tasks exhibits sen- sitivity to the off-policy ratio. 5. SAMPO 5.1. Motivation Can we derive a unified understanding of ARL training based on these insights? By systematically analyzing POs along orthogonal design dimensions in ARL, we identify key factors that determine training stability and optimiza- tion efficacy. At initialization, formatting errors and in- valid action tokens induce severe optimization noise. We eliminate these failure modes through behavior cloning and explicit format correction, constraining learning to a valid behavioral manifold. Along the importance sampling di- mension, sequence-level clipping, rather than token-wise constraints, is critical for long-horizon ARL. This mecha- nism addresses off-policy drift by suppressing harmful tra- jectories and yields substantial improvements in training stability. For advantage design, our analysis reveals that increasing advantage diversity across finer scales is essen- tial to overcoming reward sparsity. Integrating global and local signals significantly enhances credit assignment. Fi- nally, we show that dynamic trajectory filtering helps stabi- lize gradient updates by removing samples with degenerate advantages, leading to more informative policy gradients. 5.2. Our Method Guided by this unified understanding, we propose SAMPO, a new PO paradigm built on these principles. SAMPO inte- grates sequence-level clipping, fine-grained advantage es- timation, and dynamic filtering into a unified framework, yielding a stable and scalable solution for ARL. It is for- mulated as: L(θ) = 1 P N i=1 T i N X i=1 T i −1 X t=0 min s i (θ)A ′ i , clip s i (θ), 1±ε A ′ i , s.t. 0 <|y | is equivalent(a,y)| < G. (8) Here, A ′ i,k = A i + ω · A step (ˆy i,k ), s i (θ) = exp 1 |T i | P |T i |−1 t=0 log π θ (y t |x,y <t ) π θ old (y t |x,y <t ) .Across all eval- uated agentic tasks, SAMPO consistently achieves the 8 ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning strongest overall performance shown in Table 1. Com- pared to methods that modify only one dimension, SAMPO demonstrates that combining multiple design dimensions is necessary for stable and effective ARL. Notably, SAMPO delivers particularly large improvements on long-horizon interactive tasks such as ALFWorld, highlighting the im- portance of sequence-aware control in agentic settings. These results validate our central claim that stable agentic PO method requires satisfying multiple necessary condi- tions simultaneously, rather than relying on isolated algo- rithmic modifications. 5.3. Benchmarking against Inference Paradigms To further contextualize the performance of SAMPO and evaluate whether a small open-source model trained with stable RL can compete with state-of-the-art infer- ence strategies, we benchmark ARLArena against frontier closed-source models and complex multi-agent workflows. This comparison verifies a key hypothesis: principled RL training may offer greater gains in agentic tasks than heavy inference-time engineering on generic models. Experiment Setup and Results. We evaluate GPT-5.2 (OpenAI, 2025a), o3 (OpenAI, 2025b), and Gemini 2.5 Pro (Comanici et al., 2025) on ALFWorld and WebShop, under two paradigms: (i) Single LLM as Agent (SLA), following a standardized protocol; (i) Multi-Agent System (MAS), with Debate and Aggressive Debate coordination strate- gies (details are revealed in Appendix F.5). Qwen3-4B- RFT post-trained with SAMPO achieves 92.72% all-task success on ALFWorld, outperforming GPT-5.2 (51.56%) and o3-based MAS (56.25%). Open-source models with SAMPO consistently exceed larger closed-source models, showing that scale and complex inference cannot replace stable, environment-aligned ARL training. 6. Insights for Future Work Based on our systematic dissection of policy gradient de- sign choices in ARL, we identify several promising direc- tions that merit deeper exploration. (1) Clean training recipes are foundational for com- plex reasoning. ARLArena reveals that ARL is extraor- dinarily sensitive to initialization and early-stage training dynamics. A carefully constructed clean setting, com- bining short supervised cold-start SFT, format-enforcing structural constraints, and conservative KL regularization, proves essential for unlocking stable multi-turn reasoning behaviors. Without such a controlled recipe, policy gra- dient signals are easily corrupted by malformed trajecto- ries or premature collapse. This suggests that future re- search should treat training recipes not as auxiliary tricks, but as essential algorithmic components that define the feasible region in which sophisticated reasoning policies can emerge. Our codebase also provides detailed training recipes for reference. (2) IS clipping is highly sensitive, while advantage de- sign offers a comparatively stable gain. Among the policy gradient dimensions we examine, IS clipping strate- gies exhibit high sensitivity: minor changes in clipping thresholds or ratio parameterization can drastically affect stability. In contrast, advantage design tends to provide more stable but relatively modest improvements across tasks. These observations indicate that IS clipping strat- egy represents a high-risk, high-reward direction, whereas advantage design offers a more predictable but limited per- formance gains in ARL. (3) Stable ARL unlocks long-horizon scaling opportuni- ties. Once training collapse is mitigated, we observe that agentic policies can sustain performance improvements over substantially more optimization steps without degra- dation. This stability opens the door to scaling both in- teraction horizon and environment size, analogous to scal- ing laws in supervised pretraining. Consequently, future progress in the field will increasingly depend on scaling environment diversity, interaction data volume, and multi- task curricula. 7. Conclusion This work systematically analyzes how policy gradient de- sign choices impact training stability for agentic LLMs in multi-turn environments. ARLArena demonstrates that sequence-level clipping is critical for stability, while advan- tage design and dynamic filtering offer smaller but consis- tent gains, and loss aggregation has limited effect. Based on these insights, we introduce SAMPO, a unified policy optimization framework that achieves stable and effective agentic RL training. Overall, this study underscores the im- portance of principled policy design and reproducible eval- uation for advancing ARL. References Marwa Abdulhai, Isadora White, Charlie Snell, Charles Sun, Joey Hong, Yuexiang Zhai, Kelvin Xu, and Sergey Levine.Lmrl gym: Benchmarks for multi-turn re- inforcement learning with language models.arXiv preprint arXiv:2311.18232, 2023. Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, et al. Minimax-m1: Scaling test-time com- pute efficiently with lightning attention. arXiv preprint arXiv:2506.13585, 2025. 9 ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning Mingyue Cheng, Jie Ouyang, Shuo Yu, Ruiran Yan, Yu- cong Luo, Zirui Liu, Daoyu Wang, Qi Liu, and En- hong Chen. Agent-r1: Training powerful llm agents with end-to-end reinforcement learning. arXiv preprint arXiv:2511.14460, 2025. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and et al. Gemini 2.5: Pushing the frontier with advanced reason- ing, multimodality, long context, and next generation agentic capabilities, 2025. Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-group policy optimization for llm agent train- ing. arXiv preprint arXiv:2505.10978, 2025. Chang Gao, Chujie Zheng, Xiong-Hui Chen, Kai Dang, Shixuan Liu, Bowen Yu, An Yang, Shuai Bai, Jingren Zhou, and Junyang Lin. Soft adaptive policy optimiza- tion. arXiv preprint arXiv:2511.20347, 2025. Xinyan Guan, Jiali Zeng, Fandong Meng, Chunlei Xin, Yaojie Lu, Hongyu Lin, Xianpei Han, Le Sun, and Jie Zhou. Deeprag: Thinking to retrieve step by step for large language models, 2025. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shi- rong Ma, Xiao Bi, et al. Deepseek-r1 incentivizes rea- soning in llms through reinforcement learning. Nature, 645(8081):633–638, 2025. Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 sys- tem card. arXiv preprint arXiv:2412.16720, 2024. Dongfu Jiang, Yi Lu, Zhuofeng Li, Zhiheng Lyu, Ping Nie, Haozhe Wang, Alex Su, Hui Chen, Kai Zou, Chao Du, et al. Verltool: Towards holistic agentic reinforcement learning with tool use. arXiv preprint arXiv:2509.01055, 2025. Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Ser- can Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516, 2025. Devvrit Khatri, Lovish Madaan, Rishabh Tiwari, Rachit Bansal, Sai Surya Duvvuri, Manzil Zaheer, Inderjit S Dhillon, David Brandfonbrener, and Rishabh Agarwal. The art of scaling reinforcement learning compute for llms. arXiv preprint arXiv:2510.13786, 2025. Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: A comprehensive benchmark for tool-augmented llms. In Proceedings of the 2023 Con- ference on Empirical Methods in Natural Language Pro- cessing, pages 3102–3116, 2023. Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chao- fan Lin, Chen Dong, et al. Deepseek-v3. 2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556, 2025. Yinyi Luo, Yiqiao Jin, Weichen Yu, Mengqi Zhang, Srijan Kumar, Xiaoxiao Li, Weijie Xu, Xin Chen, and Jindong Wang. Agentark: Distilling multi-agent intelligence into a single llm agent. arXiv preprint arXiv:2602.03955, 2026. OpenAI. Introducing GPT-5.2, December 2025a. Ac- cessed: 2026-01-28. OpenAI. Introducing openai o3 and o4-mini, April 2025b. Accessed: 2026-01-28. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Car- roll Wainwright, Pamela Mishkin, Chong Zhang, Sand- hini Agarwal, Katarina Slama, Alex Ray, et al. Train- ing language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022. Feng Peiyuan, Yichen He, Guanhua Huang, Yuan Lin, Hanchong Zhang, Yuchen Zhang, and Hang Li. Agile: A novel reinforcement learning framework of llm agents. Advances in Neural Information Processing Systems, 37: 5244–5284, 2024. Aske Plaat, Max van Duijn, Niki van Stein, Mike Preuss, Peter van der Putten, and Kees Joost Batenburg. Agen- tic large language models, a survey.arXiv preprint arXiv:2503.23037, 2025. Timo Schick, Jane Dwivedi-Yu, Roberto Dess ` ı, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, and Nicola Cancedda. Language models can teach themselves to use tools. arXiv preprint, 2023. arXiv:2302.04761. Max-Philipp B. Schrader.gym-sokoban. https: //github.com/mpSchrader/gym-sokoban, 2018. John Schulman. Approximating kl divergence. http: //joschu.net/blog/kl-approx.html, 2017. Blog post. 10 ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithms.In Proceedings of the International Conference on Learning Representations (ICLR), 2017. arXiv:1707.06347. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In Proceedings of the Twentieth Eu- ropean Conference on Computer Systems, pages 1279– 1297, 2025. Mohit Shridhar, Xingdi Yuan, Marc-Alexandre C ˆ ot ́ e, YonatanBisk,AdamTrischler,andMatthew Hausknecht.Alfworld: Aligning text and embod- ied environments for interactive learning. arXiv preprint arXiv:2010.03768, 2020. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anand- kumar.Voyager: An open-ended embodied agent with large language models.arXiv preprint, 2023. arXiv:2305.16291. Haixin Wang, Hejie Cui, Chenwei Zhang, Xin Liu, Shuowei Jin, Shijie Geng, Xinyang Zhang, Nasser Zalmout, Zhenyu Shi, and Yizhou Sun.T 2 po: Uncertainty-guided exploration control for stable multi- turn agentic reinforcement learning.arXiv preprint arXiv:2605.02178, 2026. Jiawei Wang, Jiacai Liu, Yuqian Fu, Yingru Li, Xintao Wang, Yuan Lin, Yu Yue, Lin Zhang, Yang Wang, and Ke Wang. Harnessing uncertainty: Entropy-modulated policy gradients for long-horizon llm agents.arXiv preprint arXiv:2509.09265, 2025a. Zihan Wang, Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, et al. Ragen: Under- standing self-evolution in llm agents via multi-turn re- inforcement learning. arXiv preprint arXiv:2504.20073, 2025b. Tianxin Wei, Ting-Wei Li, Zhining Liu, Xuying Ning, Ze Yang, Jiaru Zou, Zhichen Zeng, Ruizhong Qiu, Xiao Lin, Dongqi Fu, et al. Agentic reasoning for large lan- guage models. arXiv preprint arXiv:2601.12538, 2026. Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong, Honglin Guo, Junzhe Wang, Dingwen Yang, Chenyang Liao, Xin Guo, Wei He, Songyang Gao, Lu Chen, Rui Zheng, Yicheng Zou, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, and Yu-Gang Jiang. Agentgym: Evolving large language model-based agents across diverse environments, 2024. Zhiheng Xi, Jixuan Huang, Chenyang Liao, Baodai Huang, Honglin Guo, Jiaqi Liu, Rui Zheng, Junjie Ye, Ji- azheng Zhang, Wenxiang Chen, et al.Agentgym- rl: Training llm agents for long-horizon decision mak- ing through multi-turn reinforcement learning. arXiv preprint arXiv:2509.08755, 2025. Hanchen Xia, Baoyou Chen, Zelin Zang, Yutang Ge, Guo- jiang Zhao, and Siyu Zhu. Latent poincar ́ e shaping for agentic reinforcement learning, 2026. Yutao Xie, Nathaniel Thomas, Nicklas Hansen, Yang Fu, Erran Li Li, and Xiaolong Wang.Tips: Turn- level information-potential reward shaping for search- augmented llms. In International Conference on Learn- ing Representations (ICLR), 2026. Tianshi Xu, Yuteng Chen, and Meng Li. Cleaner: Self- purified trajectories boost agentic reinforcement learn- ing. arXiv preprint arXiv:2601.15141, 2026. Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek- r1: Incentivizing reasoning capability in LLMs via re- inforcement learning. arXiv preprint arXiv:2501.12948, 2025. Zhenghai Xue, Longtao Zheng, Qian Liu, Yingru Li, Xi- aosen Zheng, Zejun Ma, and Bo An. Simpletir: End-to- end reinforcement learning for multi-turn tool-integrated reasoning. arXiv preprint arXiv:2509.02479, 2025. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chen- gen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan.Webshop: Towards scalable real-world web interaction with grounded language agents. Ad- vances in Neural Information Processing Systems, 35: 20744–20757, 2022. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023. arXiv:2210.03629. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xi- aochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gao- hong Liu, and Lingjun Liu. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025a. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xi- aochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gao- hong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025b. 11 ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning Zhaochen Yu, Ling Yang, Jiaru Zou, Shuicheng Yan, and Mengdi Wang.Demystifying reinforcement learning in agentic reasoning. arXiv preprint arXiv:2510.11701, 2025c. Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Ke- qing He, Zejun Ma, and Junxian He.Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild.arXiv preprint arXiv:2503.18892, 2025. Hanchen Zhang, Xiao Liu, Bowen Lv, Xueqiao Sun, Bohao Jing, Iat Long Iong, Zhenyu Hou, Zehan Qi, Hanyu Lai, Yifan Xu, et al. Agentrl: Scaling agentic reinforcement learning with a multi-turn, multi-task framework. arXiv preprint arXiv:2510.04206, 2025. Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimiza- tion. arXiv preprint arXiv:2507.18071, 2025. Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neu- big. Webarena: A realistic web environment for building autonomous agents, 2024. Yifei Zhou, Song Jiang, Yuandong Tian, Jason Weston, Sergey Levine, Sainbayar Sukhbaatar, and Xian Li. Sweet-rl: Training multi-turn llm agents on collabora- tive reasoning tasks. arXiv preprint arXiv:2503.15478, 2025. 12 ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning Supplementary Materials for ARLArena A. More Details on Research Dimension A.1. Loss Aggregation As discussed in Section 2.2, the policy gradient objective for agentic LLMs is implemented through a batch-level loss aggregation over token-level surrogate losses. For a batch of N sampled trajectories y i N i=1 , where trajectory i has length T i , we define the token-level loss as ℓ i,t (θ) := min(w i,t (θ)A i , clip(w i,t (θ), 1− ε, 1 + ε)A i ), (9) where w i,t (θ) = π θ (y i,t | x i ,y i,<t )/π θ old (y i,t | x i ,y i,<t ) and A i denotes the (sequence-level) advantage associated with trajectory y i . Different loss aggregation strategies correspond to differ- ent empirical estimators of the expectation over trajectories and tokens. Below we summarize several commonly used schemes. Token-mean. The token-mean estimator averages the loss uniformly over all unmasked tokens in the batch: L token-mean (θ) = 1 P N i=1 T i N X i=1 T i −1 X t=0 ℓ i,t (θ).(10) This scheme assigns equal weight to each token across the entire batch and is invariant to trajectory length at the se- quence level. Token-mean has been adopted in several re- cent works (e.g., DAPO) as a means of stabilizing opti- mization. However, because trajectories with longer re- sponses contribute more tokens, they implicitly receive larger total weight, which may bias optimization toward long trajectories. Sequence-mean token-mean (Seq-mean-token-mean). This estimator first averages over tokens within each tra- jectory and then averages across trajectories: L seq-mean-token-mean (θ) = 1 N N X i=1 1 T i T i −1 X t=0 ℓ i,t (θ).(11) Under this scheme, each trajectory contributes equally re- gardless of its length. Equivalently, each token is weighted by 1/T i . As a result, shorter trajectories assign larger per-token weight, while longer trajectories are relatively down-weighted. This behavior can introduce response- level length bias, rewarding short correct trajectories more strongly and penalizing long incorrect trajectories less. Sequence-meantoken-sum(Seq-mean-token-sum). An alternative aggregation removes the per-trajectory normalization over tokens: L seq-mean-token-sum (θ) = 1 N N X i=1 T i −1 X t=0 ℓ i,t (θ).(12) This formulation corresponds to maximizing the expected cumulative surrogate objective over full trajectories. Com- pared to Seq-mean-token-mean, longer trajectories receive proportionally larger weight. Sequence-mean token-sum with length normalization (Seq-mean-token-sum-norm). In practice, some imple- mentations normalize by a fixed maximum generation length T max : L seq-mean-token-sum-norm (θ) = 1 NT max N X i=1 T i −1 X t=0 ℓ i,t (θ). (13) This estimator enforces a uniform upper bound on the con- tribution of each trajectory and assigns equal weight to to- kens across batches under a fixed-length budget. Discussion. These aggregation schemes differ primarily in how they trade off trajectory-level fairness, token-level weighting, and variance control. Seq-mean-token-mean and token-mean are the two most commonly used estima- tors in practice and are the focus of our empirical analysis in Section 4.4. The remaining variants are included here for completeness and to clarify their implicit inductive biases in agentic reinforcement learning. A.2. Importance Sampling Clipping As discussed in Section 2.2, importance sampling (IS) clip- ping plays a central role in stabilizing off-policy policy op- timization. While all methods considered in this work rely on the same token-level importance ratio w i,t (θ) = π θ (y i,t | x i ,y i,<t ) π θ old (y i,t | x i ,y i,<t ) ,(14) they differ substantially in where and how clipping is ap- plied. Below we summarize the clipping mechanisms of GRPO, CISPO, SAPO, and GSPO. A.2.1. GRPO Group Relative Policy Optimization (GRPO) adopts the standard PPO-style hard clipping applied independently at each token: ℓ GRPO i,t (θ) = min(w i,t (θ)A i , clip(w i,t (θ), 1− ε, 1 + ε)A i ). (15) 13 ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning Clipping is performed directly on the token-level impor- tance ratio. When w i,t falls outside the clipping range, the gradient contribution of that token is truncated. A.2.2. CISPO ClippedImportanceSamplingPolicyOptimization (CISPO) modifies GRPO by clipping the importance ratio itself rather than the surrogate objective. Specifically, the clipped ratio is defined as ̃w i,t (θ) = ( 1 + ε, w i,t (θ) > 1 + ε, w i,t (θ), otherwise, (16) and is treated as a stop-gradient quantity. The resulting loss takes the form ℓ CISPO i,t (θ) = sg( ̃w i,t (θ)) A i logπ θ (y i,t | x i ,y i,<t ), (17) where sg(·) denotes the stop-gradient operator. By avoid- ing hard truncation of token updates, CISPO preserves gra- dient flow for clipped tokens while still bounding their in- fluence. However, clipping remains token-local and does not explicitly enforce sequence-level coherence. A.2.3. SAPO Soft Adaptive Policy Optimization (SAPO) replaces hard clipping with a smooth, temperature-controlled gating function. The surrogate loss is defined as ℓ SAPO i,t (θ) = f i,t (w i,t (θ))A i ,(18) where f i,t (x) = σ τ i,t (x−1) · 4 τ i,t , τ i,t = ( τ pos , A i > 0, τ neg , A i < 0. (19) Here σ(·) denotes the sigmoid function. SAPO imple- ments a continuous trust region: near on-policy updates are preserved, while off-policy updates are smoothly atten- uated rather than abruptly clipped. The asymmetric tem- perature design further suppresses high-variance negative- advantage updates. Despite improved smoothness, SAPO remains a token-level method and does not explicitly pre- vent a few extreme tokens from destabilizing a full trajec- tory. A.2.4. GSPO Group Sequence Policy Optimization (GSPO) fundamen- tally changes the unit of clipping by operating at the se- quence level. The sequence-level importance ratio is de- fined as s i (θ) = exp 1 T i T i −1 X t=0 logw i,t (θ) ! = π θ (y i | x i ) π θ old (y i | x i ) 1/T i . (20) Clipping is then applied once per sequence: ℓ GSPO i (θ) = min(s i (θ)A i , clip(s i (θ), 1− ε, 1 + ε)A i ). (21) All tokens within a trajectory share the same clipped up- date. This design aligns the unit of importance sampling with the unit of reward and enforces strong sequence-level coherence. As a result, GSPO effectively suppresses high- variance token outliers and yields substantially more stable optimization in long-horizon agentic reinforcement learn- ing. Summary. In summary, GRPO, CISPO, and SAPO ap- ply clipping at the token level with increasing degrees of smoothness, whereas GSPO performs clipping at the se- quence level. Our empirical results in Section 4.1 demon- strate that sequence-level clipping is a key factor for stabi- lizing multi-turn agentic RL training. A.3. Advantage Design This section provides detailed formulations of the advan- tage designs introduced in Section 2.2, including Group- in-Group Policy Optimization (GiGPO) and Entropy- Modulated Policy Gradients (EMPG). Both methods ex- tend standard group-based advantage estimation to better handle long-horizon agentic reinforcement learning. Notation. We consider a batch of N trajectoriesτ i N i=1 , where each trajectory τ i = (s i,k ,a i,k ,r i,k ) K i k=1 is gener- ated under the behavior policy π θ old . The total return of a trajectory is denoted by R(τ i ) = T i X t=1 r i,k .(22) A.3.1. GROUP-IN-GROUP POLICY OPTIMIZATION (GIGPO) GiGPO introduces a hierarchical advantage structure that combines trajectory-level and step-level relative advan- tages. The design preserves the critic-free and group-based nature of GRPO while enabling finer-grained credit assign- ment. Episode-level relative advantage. GiGPO first com- putes a trajectory-level (episode-level) relative advantage by normalizing total returns within the rollout group: A i = R(τ i )− mean R(τ j ) N j=1 F norm R(τ j ) N j=1 ,(23) where F norm (·) is a normalization factor. In the original formulation, F norm may be chosen as the standard devia- tion or a fixed constant. 14 ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning Step-level relative advantage via anchor state grouping. To assign fine-grained credit within a trajectory, GiGPO constructs step-level groups based on repeated environment states. Let U denote the set of distinct environment states appearing in the trajectory batch. For each anchor state ̃s∈U , a step-level group is defined as G S ( ̃s) = a i,k ,R i,k s i,k = ̃s ,(24) where R i,k denotes the discounted return from step k re- ward: R i,k = T i X m=t γ m−t r i,m .(25) Within each step-level group, GiGPO computes a relative advantage for individual actions: A step (ˆy i,k ) = R i,k − mean (R j,k ′ | (a j,k ′ ,R j,k ′ )∈G S ( ̃s)) F norm (R j,k ′ | (a j,k ′ ,R j,k ′ )∈G S ( ̃s)) . (26) Combined advantage. The final advantage used for pol- icy optimization is a linear combination of episode-level and step-level components: A ′ i,k = A i + ω A step (y i,k ),(27) where ω ≥ 0 is a weighting coefficient controlling the con- tribution of step-level credit. A.3.2. ENTROPY-MODULATED POLICY GRADIENTS (EMPG) Entropy-Modulated Policy Gradients (EMPG) augments the advantage function by incorporating step-wise uncer- tainty measured via policy entropy. The method reshapes the learning signal at each decision step while preserving a trajectory-level optimization objective, making it suitable for long-horizon agentic reinforcement learning. Step-level entropy. For a trajectory τ i and its t-th step, EMPG defines a step-level entropy H i,t as the average token-level entropy over the tokens generated at that step: H i,t =− 1 |y i,t | |y i,t | X j=1 X v∈V π θ (v | y i,t,<j ) logπ θ (v | y i,t,<j ), (28) where |y i,t | is the number of tokens in step t, y i,t,<j de- notes the prefix before token j within that step, and V is the vocabulary. Entropy-modulated advantage. Let A(τ i ) denote the trajectory-level advantage (e.g., computed via group-based normalization as described in Section 2.2). EMPG defines a step-wise modulated advantage as A mod (i,t) = g(H i,t ) A(τ i ) + ζ f(H i,t+1 ),(29) where g(·) is a self-calibrating scaling function based on current-step entropy, f(·) is a future-clarity bonus depend- ing on the next step, and ζ ≥ 0 controls the contribution of the future-clarity term. Self-calibrating gradient scaling. The scaling function g(·) reweights the trajectory-level advantage according to the relative entropy of each step within a batch: g(H i,t ) = exp −k ̃ H i,t 1 P j T j P j,t ′ exp −k ̃ H j,t ′ ,(30) where ̃ H i,t denotes a batch-normalized entropy value, T j is the length of trajectory τ j , and k > 0 is a temperature pa- rameter. This normalization ensures that the average scal- ing factor over the batch equals one. Future clarity bonus. To encourage transitions toward lower-uncertainty future states, EMPG introduces a future- clarity bonus defined as f(H i,t+1 ) = exp −k ′ ̃ H i,t+1 ,(31) where k ′ > 0 controls sensitivity to the entropy of the next step. Finaladvantagenormalization. Aftercomputing A mod (i,t) for all steps in the batch, EMPG applies a final batch-level normalization (e.g., zero-mean normalization) before using the resulting advantages in policy gradient updates. B. Key Hyper-parameter The hyperparameters reported in Table 6 are determined through task-specific grid search.For each policy op- timization method and environment, we sweep over the method-relevant hyperparameters while keeping the re- maining training and optimization settings fixed. The fi- nal configurations correspond to the stable settings selected from the grid search. C. Additional Experiment Result C.1. Performance on 8B Model To further investigate the scalability of our findings, we evaluate the 8B parameter model (Qwen3-8B) on ALF- World, which serves as a representative benchmark for complex, multi-turn agentic tasks. Given the substantial computational requirements for large-scale RL training, we focus on this environment to verify if the core design prin- ciples distilled from the 4B models remain consistent at a larger scale. 15 ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning Table 6. Key training hyperparameters for agentic RL experiments across four tasks (ALFWorld, WebShop, Sokoban, TIR Math). “–” indicates the method is not applicable to that task. CategoryALFWorldWebShopSokobanTIR Math Model and Environment Configuration Base modelQwen3-4B-RFTQwen3-4B-RFTQwen3-4B-VL-Instruct-RFTQwen3-4B-Base Max interaction steps5015155 Memory context window2 (turns)2 (turns)2 (turns)8196 (tokens) Group rollout size8885 Max prompt length2048409610248196 Max response length5125125124096 Format penalty coefficient0.10.10.10.1 Training Optimization Group normalization modemeanstdnormmeanstdnormmeanstdnormmeanstdnorm Learning rate1× 10 −6 1× 10 −6 1× 10 −6 1× 10 −6 Mini-batch size25612864128 KL coefficient0.010.010.010 Rollout and Inference Configuration Rollout enginevLLMvLLMvLLMvLLM Temperature (training)1.01.01.01.0 Temperature (validation)0.60.60.70.6 Top-p (validation)0.950.950.950.95 Top-k (validation)20202020 Training and Batching User Prompt Number161632512 Validation batch size128128128128 Total epochs200(∼ 24h)200(∼ 22h)200(∼ 12h)17(∼ 60h) GPUsNVIDIA H200/B200NVIDIA H200/B200NVIDIA H200/B200NVIDIA H200/B200 PO-specific Parameters GRPO ε high 0.20.20.20.28 ε low 0.20.20.20.2 GIGPO ε0.20.20.2– γ0.950.950.95– ω111– EMPG ε0.20.20.2– k,k ′ 1.01.01.0– ζ0.050.050.05– GSPO ε high 4e-34e-24e-34e-4 ε low 3e-33e-23e-33e-4 CISPO ε high 0.20.20.20.28 ε low 1111 SAPO τ pos 1.01.01.01.0 τ neg 1.051.051.051.05 DAPO ε high 0.20.20.20.28 ε low 0.20.20.20.2 N oversample 3333 16 ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning As shown in Table 7, the experimental results on ALF- World and WebShop demonstrate that the relative per- formance gains and stability trends are highly consistent with our observations in the 4B experiments in Section 4. Specifically, the critical importance of sequence-level clip- ping is reaffirmed: even with increased model capacity, it remains the indispensable factor for preventing training collapse. Furthermore, we observe that the benefits of ad- vantage design and dynamic filtering persist at this larger scale, providing consistent but incremental improvements to final performance. In contrast, the choice of loss ag- gregation continues to exhibit limited impact, echoing our findings on 4B models. These results collectively suggest that the hierarchical impact of policy design choices—and the resulting SAMPO recipe—is robust and scale-invariant, effectively leveraging the enhanced reasoning capabilities of larger models while maintaining stable training dynam- ics. C.2. Additional Analysis Result We further visualize the training dynamics of CISPO and CISPO SM on the AlfWorld task using diagrams. Specif- ically, following the same setup as in the main text, we categorize trajectories according to three factors: the sign of the advantage, whether the entropy exceeds a prede- fined threshold, and whether the IS ratio is greater than zero. These criteria partition the samples into eight groups, which we use to analyze how the KL divergence evolves during training. Consistent with our earlier findings, we clearly observe that after CISPO collapses, trajectories with negative ad- vantages and low IS ratios (i.e., adv < 0 and IS < 1) rapidly dominate the distribution. This imbalance corre- lates strongly with the surge in KL divergence and subse- quent training instability. This observation also explains why CISPO SM , which in- corporates sequence-level masking, achieves substantially improved stability: by masking these harmful negative- advantage and low-ratio trajectories, the optimization pro- cess avoids pathological updates and maintains more bal- anced gradient signals. C.3. Task Environment Details ALFWorld (Shridhar et al., 2020): It provides a text- based interactive setting in which LLM agents are required to complete goal-driven tasks that involve reasoning over multiple sequential decisions. The environment focuses on everyday household activities and evaluates an agent’s abil- ity to plan and act through iterative interaction. WebShop (Yao et al., 2022): It is a large-scale interac- tive environment that places agents in realistic e-commerce scenarios, requiring them to interpret user instructions and make sequential decisions to identify and purchase suitable products. Sokoban (Schrader, 2018): It is a classic grid-based plan- ning task where an agent navigates a 2D environment to push all boxes onto designated target cells. The state is rep- resented visually, and the agent selects from discrete move- ment actions. TIR Math (Xue et al., 2025): This task focuses on stan- dard mathematical question answering, where Python is used as a tool for intermediate calculations and symbolic reasoning. The overall pipeline follows Xue et al. (2025). The training data are adapted from SimpleRL (Zeng et al., 2025), and evaluation is conducted on the AIME and AIME25 benchmarks.Performance is measured using pass@k, following the evaluation protocol in Yu et al. (2025b). D. Related Work Large language models have demonstrated strong capabil- ities in agent-based environments and attracted increas- ing attention (Yao et al., 2022; Shridhar et al., 2020; Li et al., 2023). Prior studies investigate LLMs as agents in multi-turn, action-based environments, emphasizing long- horizon memory and explicit tool use for sequential deci- sion making and reasoning (Yao et al., 2023; Schick et al., 2023; Wang et al., 2023). Recently, driven by the success of reinforcement learning in reasoning (Xu et al., 2025; Ope- nAI, 2025a; Khatri et al., 2025), RL has been extended to agentic settings (Jin et al., 2025; Plaat et al., 2025; Ab- dulhai et al., 2023; Yu et al., 2025c). Several representa- tive RL frameworks for LLM agents have emerged. AG- ILE (Peiyuan et al., 2024) proposes a framework for LLM- driven conversational agents capable of planning, tool use, and expert consultation. SWEET-RL (Zhou et al., 2025) studies collaborative LLM agents that interact with simu- lated human partners in ColBench, where agents ask clari- fying questions and learn from multi-turn feedback. Agent- R1 (Cheng et al., 2025) extends this paradigm to exter- nal tool-based environments and enables multi-turn reason- ing with tool calls. Similarly, AgentGym-RL (Xi et al., 2025) presents an RL framework for autonomous LLM agents that supports multi-turn interactions, modular ar- chitectures, and real-world scenarios. AgentRL (Zhang et al., 2025) develops a multi-turn, multi-task RL system and demonstrates superior performance relative to closed- source models. VerlTool (Jiang et al., 2025) focuses on tool-using LLM agents and aligns well with the VeRL codebase. Most prior work provides limited analysis of agentic RL training instability. In contrast, ARLArena of- fers a unified training and analysis framework for examin- ing how policy-gradient design choices relate to stability 17 ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning DimensionMethod ALFWorldWebShop Task ScoreSuccess RateTask ScoreSuccess Rate BaseGRPO2.3750.9285.4873.98 Loss AggGRPO ST 1.68↓ 29.1%49.31↓ 3.2%91.21↑ 6.7%83.57↑ 13.0% Importance Sampling SAPO0.08↓ 96.6%1.93↓ 96.21%84.73↓ 0.9%74.47↑ 0.7% CISPO0.80↓ 66.2%30.83↓ 39.5%87.80↑ 2.7%73.74↓ 0.3% GSPO5.05↑ 113.1%79.70↑ 56.5%91.61↑ 7.2%83.15↑ 12.4% Advantage Design GIGPO4.10↑ 73.0%80.03↑ 57.2%89.26↑ 4.4%78.91↑ 6.7% EMPG4.51↑ 90.3%71.48↑ 40.4%88.60↑ 3.6%75.46↑ 2.0% Dynamic Sampling DAPO GRPO 0.81↓ 65.8%38.11↓ 25.16%86.52↑ 1.2%76.52↑ 3.4% DAPO GIGPO 2.49↑ 5.1%60.27↑ 18.4%91.92↑ 7.5%82.42↑ 11.4% OursSAMPO8.98↑ 278.9%97.71↑ 91.9%93.43↑ 9.3%84.02↑ 13.6% Table 7. Performance on SFT version of Qwen3-8B for ALFWorld and WebShop correspondingly. The overall trend on the 8B variant remains consistent, and SAMPO continues to achieve the best performance, indicating stable gains under model scaling. Figure 4. Sequence-Level IS Analysis of CISPO and CISPO SM (CISPO with sequence masking) on ALFWorld. and performance across agentic tasks. E. Another Roadmap of Building Agentic LLM: Multi-agent System E.1. Debate LetA = A 1 ,A 2 ,...,A N denote the set of N agents, where N is an odd integer to prevent tie-breaking scenar- ios during majority voting. Let x denote the task prompt. In the initial round (t = 0), each agentA i independently gen- erates a candidate solution c (0) i based solely on the prompt x: c (0) i =A i (x), ∀i∈1,...,N(32) LetC (t) = c (t) 1 ,c (t) 2 ,...,c (t) N be the set of candidate so- lutions at round t. We define a majority consensus function M(·) that returns the solution y if it appears in more than half of the agent responses: y =M(C (t) ) = ( ˆcif c∈C (t) : c = ˆc > N 2 ∅ otherwise (33) IfM(C (0) )̸=∅, the process terminates and outputs y. Oth- erwise, the system enters the debate phase. The process iterates through debate rounds t = 1, 2,...,T max . For each round, we construct the debate prompt for each agent, which includes the original prompt x, the set of unique can- didate solutions from the previous round Unique(C (t−1) ), and agents’ reasoning in previous round R (t−1) .Let R (t) = r (t) 1 ,r (t) 2 ,...,r (t) N be the agents’ reasoning at round t, andR (0) =∅. r (t) i ,c (t) i =A i x, Unique(C (t−1) ),R (t−1) .(34) At the end of each round t, we check for consensus again and output the solution y if consensus is reached. This mechanism enables agents to either rectify perceived flaws by proposing a new solution or align with a peer by voting for an existing candidate. The debate terminates when a 18 ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning majority consensus is achieved,M(C (t) ) ̸= ∅. If the max- imum iteration limit T max is reached without consensus, the final output y is randomly sampled from the final set of candidatesC (T max ) . E.2. Aggressive Debate We extend the Debate framework discussed above to build a decisively goal-oriented variant designed to prioritize task completion over exhaustive exploration. While the stan- dard framework seeks consensus on an optimal solution, the aggressive variant compels agents to accept partial suc- cess by securing the best available option within a strict finite horizon. Formally, we modify the agent A i by conditioning it on an additional constraint set I agg .Unlike standard de- bate agents that aim for a perfect solution, the aggres- sive agent A i (·|I agg ) operates under a modified utility function characterized by several governing principles: (1) Bounded Exploration: The agent must finalize the inter- action within a finite horizon. This constraint suppresses excessive exploration and ensures the agent commits to a definitive outcome rather than prolonging the information- gathering phase; (2) Temporal Efficiency: The agent is encouraged to conclude the interaction as early as possi- ble; (3) Incentive Awareness: The agent is explicitly in- formed that partial rewards are available. This awareness incentivizes the agent to accept high-utility suboptimal out- comes when a perfect solution is unattainable; (4) Prag- matic Optimization: The agent prioritizes securing a re- sult that maximizes available partial rewards rather than seeking a theoretical global optimum, thereby avoiding di- minishing returns associated with perfecting the solution in complex environments. E.3. Experiment Results on SLA and MAS F. Failure Analysis F.1. Method: Sankey Graphs for Action-Transition Flows We analyze agent rollouts by visualizing step-wise action transitions with Sankey graphs. Each column corresponds to a time step, node height indicates the empirical fre- quency of an action at that step, and edges represent tran- sitions between consecutive steps. Compared with action histograms, Sankey graphs preserve temporal structure and thus reveal loop-like behaviors (e.g., repetitive pagination or oscillation between two actions) that dominate long- horizon failures. F.2. WebShop: Action-Transition Patterns and Failure Modes Overall flow (API agent). The API agent is a single- agent baseline powered by GPT-4o via API under the same interaction protocol, without any task-specific train- ing. Figure 5 summarizes WebShop trajectories of the API agent, where green links correspond to successful episodes and red links correspond to failures. A large fraction of fail- ures is characterized by repetitive next actions, suggest- ing exploration inefficiency where the agent keeps paginat- ing without making progress toward constraint satisfaction. Failure-only flow with action coloring. Figure 6 fo- cuses on failed trajectories and colors nodes by action type. Two dominant failure patterns are observed: (i) Pagination loops: long runs of next (and occasional search) that rarely transition into click product (product-detail in- spection); (i) Backtracking oscillation: frequent alterna- tion between click product and back, suggesting re- peated revisits to previously viewed product pages and lim- ited progress toward constraint satisfaction. Notably, our API agent is provided with a long interaction history (past actions and observations) in the prompt, so this pattern is unlikely to be explained by insufficient context alone. In- stead, it may reflect limited effective memory usage: with- out structured tracking or summarization of verified at- tributes and visited items, the agent may fail to retrieve previously established evidence from a long, unstructured context and thus re-check similar products. We emphasize that this is only one plausible factor; we find instruction ambiguity or conflicting constraints may also contribute. F.3. WebShop: How RL Post-training Changes Behaviors Overall flow (RL-optimized agent). Figure 7 shows the same visualization for our RL-optimized agent (post- trained with RL). Compared with the API baseline, the RL agent exhibits fewer next-dominated failure paths and a higher proportion of trajectories that transition into click product and eventually attempt buy, consistent with more targeted product inspection and earlier decision making. Remaining failure modes after RL post-training. Fig- ure 8 focuses on failed RL trajectories. While next- heavy pagination loops become less prominent, two resid- ual issues remain: (i) Backtracking-heavy browsing: re- peated click other/back transitions, suggesting inef- ficient navigation; (i) Premature purchase: occasional buy attempts that do not satisfy all constraints, suggesting incomplete constraint tracking. 19 ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning Method ALFWorldWebShop PickLookCleanHeatCoolPick2AllScoreSuccess GPT-4o61.1133.3336.3650.0045.4563.6450.0013.6012.50 GPT-5.270.0366.0735.3762.3052.0837.3651.5626.5626.56 Debate67.7464.2833.3360.0052.3865.0056.2534.6522.65 Aggressive Debate–61.5328.51 Gemini-2.5-pro84.9761.6163.9422.2262.5075.2566.41– GRPO87.4162.6546.4272.2858.8938.3772.6175.3257.71 SAPO34.4932.1924.1324.9216.219.3725.1673.8552.10 CISPO76.0337.1258.5650.9757.8823.6854.4267.9654.71 GSPO90.3679.3190.7175.4577.9548.9578.6185.2972.48 GIGPO94.8083.0386.3781.1575.3859.2181.0967.7656.55 EMPG84.1861.5369.8372.4946.510.0457.9179.1664.32 DAPO GRPO 81.2837.5753.9740.1651.286.4349.5862.4346.17 DAPO GIGPO 85.0455.2665.3558.9856.5226.5760.5588.1076.82 SAMPO96.3088.4993.6592.4292.7088.3592.7288.0474.08 Table 8. Unified comparison across ALFWorld (six task types + overall) and WebShop (score and success rate). The upper block reports closed-source baselines and multi-agent strategies; the lower block reports policy optimization methods trained with Qwen3-4B. Figure 5. WebShop action-transition Sankey for the API agent. Green flows denote successful trajectories and red flows denote failures. F.4. ALFWorld: Action-Transition Patterns Figure 9 visualizes ALFWorld rollouts.Navigation actions (e.g., go, look) dominate early steps across episodes, whereas successful trajectories more often tran- sition into object-centric interactions (e.g., examine, open/close, take, use) and explicit state-checking (inventory). In contrast, failed trajectories frequently exhibit prolonged navigation with comparatively fewer ob- ject interactions, which may reflect weak progression to- ward concrete object-level subgoals and imperfect tracking of what has already been tried or collected over long hori- zons. F.5. Implications Our analysis suggests two actionable directions:(1) Loop-aware control (e.g., detecting repeated next or click product ↔ back cycles and triggering a plan change); (2) Explicit constraint/state memory (e.g., in- troducing a lightweight memory agent that maintains a con- cise record of visited items and verified constraints, and feeds the acting agent with short summaries or retrieval re- sults). Together, these mechanisms may further improve robustness beyond RL post-training. G. Visualization G.1. Evidence of Format v.s. Dynamic Filtering To support the analysis in Section 4.4, we report the format validity ratio during training for different policy optimiza- tion variants. The results illustrate that DAPO combined with GIGPO maintains more stable format behavior than DAPO+GRPO after dynamic filtering. 20 ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning Figure 6. WebShop failure-only action-transition Sankey for the API agent. Nodes are colored by action type (e.g., search, clickproduct, clickother, buy, back, next). H. Case Study H.1. Prompt Templates H.1.1. TIR MATH TIR Math Prompt TIR_TEMPLATE = """ Solve the following problem step by step . You now have the ability to selectively write executable Python code to enhance your reasoning process. The Python code will be executed by an external sandbox, and the output (after "Code execution result: ") is returned to aid your reasoning and help you arrive at the final answer. The Python code should be complete scripts, including necessary imports. Code Format: Each code snippet is wrapped between triple backticks. You need to use ‘print()‘ to output intermediate results. Answer Format: You can use the ‘final_answer()‘ function in the code to return your final answer. For example, to answer the User Question : What is the result of the 5 + 3 + 1294.678?, you can write: answer = 5 + 3 + 1294.678 final_answer(answer) You can also use to return your answer. The last part of your response should be : ’The final answer goes here.’ User Question: """ H.1.2. WEBSHOP WebShop Prompt WEBSHOP_TEMPLATE = """ You are an expert autonomous agent operating in the WebShop e-commerce environment. Your task is to: task_description. Prior to this step, you have already taken step_count step(s). Below are the most recent history_length observations and the corresponding actions you took: action_history You are now at step current_step and your current observation is: current_observation. Your admissible actions of the current situation are: [ available_actions ]. 21 ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning Figure 7. WebShop action-transition Sankey for the RL-optimized agent. Green flows denote successful trajectories and red flows denote failures. Now it’s your turn to take one action for the current step. You should first reason step-by-step about the current situation, then think carefully which admissible action best advances the shopping goal. This reasoning process MUST be enclosed within <think> </think> tags. Once you’ve finished your reasoning, you should choose an admissible action for the current step and present it within <action> </action> tags. """ H.1.3. ALFWORLD ALFWorld Prompt ALFWORLD_TEMPLATE = """ You are an expert agent operating in the ALFRED Embodied Environment. Your task is to: task_description Prior to this step, you have already taken step_count step(s). Below are the most recent history_length observations and the corresponding actions you took: action_history You are now at step current_step and your current observation is: current_observation Your admissible actions of the current situation are: [admissible_actions]. Now it’s your turn to take an action. You should first reason step-by-step about the current situation. This reasoning process MUST be enclosed within <think> </think> tags. Once you’ve finished your reasoning, you should choose an admissible action for the current step and present it within <action> </action> tags. """ H.1.4. SOKOBAN Sokoban Prompt SOKOBAN_TEMPLATE = """ You are an expert agent operating in the Sokoban environment. Your goal is to push all the boxes onto the target spots. Once all boxes are on the targets, you win! # Rules You can only push boxes. You can’t pull them, so plan ahead to avoid getting stuck. You can’t walk through or push boxes into walls. To avoid traps, do not push boxes into corners or against walls where they can’t be moved again. 22 ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning Figure 8. WebShop failure-only action-transition Sankey for the RL-optimized agent. Nodes are colored by action type (e.g., search, clickproduct, clickother, buy, back, next). # Visual Elements in the Image: Character: A small, green alien-like figure with two antennae and black eyes. It represents you. Box: A yellow crate marked with an orange "X" across its front. It is the box you need to push. Target: A black tile outlined in red, with a small red diamond shape in the center. It marks the destination where a box should be pushed. # Current Step Your current observation is shown in the image: <image> Your admissible actions are ["up", "down ", "left", "right"]. Now it’s your turn to make a move ( choose ONE action only for the current step). You should first reason step-by-step about the current situation - observe the positions of boxes and targets, plan a path to push a box toward a target, and avoid traps like corners or walls. This reasoning process MUST be enclosed within <think> </think> tags. Once you’ve finished your reasoning, you should choose an admissible action for the current step and present it within <action> </action> tags. """ H.2. Multi-turn State-Action Templates H.2.1. TIR MATH TIR Math Rollout Example Input: Let a, b, and c be real numbers such that a >= b >= c >= 0 and a + b = 16. Find the maximum value of sqrt(c(b - c)) + sqrt(c(a - c)). Output: To solve this problem step by step: 1. Define the constraints and variables. 2. Use Python to iterate through possible values of a, b, and c. 3. Calculate the expression for each set of values. 4. Keep track of the maximum value found during iterations. 5. Return the maximum value. Let’s implement this approach: 1 import numpy as np 2 3 # Define the range and step size for a and b since we know they sum to 16 4 a_vals = np.arange(0, 17, 0.01) 5 b_vals = np.arange(0, 17, 0.01) 6 7 max_value = -np.inf 8 23 ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning Figure 9. ALFWorld action-transition Sankey diagrams for the API agent. Top: Success (green) vs. failure (red) trajectories. Bottom: Failure trajectories with nodes colored by action type. 9 # Iterate through possible values of a and b 10 for a in a_vals: 11 for b in b_vals: 12 if a + b == 16 and a >= b: 13 # Iterate through possible values of c 14 for c in np.arange(0, min(b, a), 0.01): 15 # Check if this configuration satisfies all constraints 16 if c <= b and c <= a and a >= b >= c: 17 # Calculate the expression 18 expr_val = np. sqrt(c * (b - c)) + np.sqrt(c * (a - c)) 19 # Update the maximum value if necessary 20 if expr_val > max_value: 21 max_value = expr_val 22 23 # Print the maximum value found 24 print(f"Maximum Value: max_value") Code execution result: Maximum Value: 8.0 8 24 ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning AlfWorldWebShop Figure 10. Format validity ratio during training on AlfWorld and WebShop for GRPO, GIGPO, DAPO GRPO , and DAPO GIGPO . Applying dynamic filtering to GRPO leads to degraded format stability, whereas DAPO GIGPO maintains stable format behavior across training. H.2.2. WEBSHOP WebShop Rollout Example (one turn) Input: user You are an expert autonomous agent operating in the WebShop e-commerce environment. Your task is to: Find me machine washable men’s t-shirts with short sleeve for tumble dry with color: 36 pack mix, and size: x-large, and price lower than 160.00 dollars. Your current observation is: ’Search’. Your admissible actions of the current situation are: [ ’search[<your query>]’, ’click[search]’, ]. Now it’s your turn to take one action for the current step. You should first reason step-by-step about the current situation, then think carefully which admissible action best advances the shopping goal. This reasoning process MUST be enclosed within <think> </think> tags. Once you’ve finished your reasoning, you should choose an admissible action for the current step and present it within <action> </action> tags. Output: <think> Okay, so I need to find machine washable men’s t-shirts, short sleeve, for tumble dry, 36 pack mix, x-large, under $160. The current observation is ’Search’, so I need to create a specific query that includes all these parameters. </think> <action> search[men’s short sleeve t-shirts, 36 pack mix, x-large, machine washable , tumble dryable, color: 36 pack mix, price: $160 or less] </action> H.2.3. ALFWORLD ALFWorld Rollout Example (one turn) Input: user You are an expert agent operating in the ALFRED Embodied Environment. Your current observation is: -= Welcome to TextWorld, ALFRED! =- You are in the middle of a room. Looking quickly around you, you see a cabinet 13, a cabinet 12, a cabinet 11, a cabinet 10, a cabinet 9, a cabinet 8, a cabinet 7, a cabinet 6, a cabinet 5, a cabinet 4, a cabinet 3, a cabinet 2, a cabinet 1, a coffeemachine 1, a countertop 1, a diningtable 1, a drawer 1, a fridge 1, a garbagecan 1, a microwave 1, 25 ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning a shelf 3, a shelf 2, a shelf 1, a sinkbasin 1, a stoveburner 4, a stoveburner 3, a stoveburner 2, a stoveburner 1, and a toaster 1. Your task is to: put a cool egg in microwave. Your admissible actions of the current situation are: [ ’go to cabinet 1’, ..., ’go to cabinet 13’, ’go to coffeemachine 1’, ’go to countertop 1’, ’go to diningtable 1’, ’go to drawer 1’, ’go to fridge 1’, ’go to garbagecan 1’, ’go to microwave 1’, ’go to shelf 1’, ’ go to shelf 2’, ’go to shelf 3’, ’go to sinkbasin 1’, ’go to stoveburner 1’, ..., ’go to stoveburner 4’, ’go to toaster 1’, ’inventory’, ’look’ ]. Now it’s your turn to take an action. You should first reason step-by-step about the current situation. This reasoning process MUST be enclosed within <think> </think> tags. Once you’ve finished your reasoning, you should choose an admissible action for the current step and present it within <action> </action> tags. Output: <think> Okay, I need to put a cool egg in the microwave. First, where would I find a cool egg? The fridge is typically where you keep cool items like eggs. So maybe I should start by going to the fridge. Looking at the list of admissible actions, there’s "go to fridge 1". That’s probably the first step. Once there, I can check the interior of the fridge to see if there’s an egg. Alternatively, maybe I should check the dining table or the countertops in case the egg is already on a surface. But I think the fridge is more likely to have an egg. Since the task is to put a cool egg in the microwave, it’s probably in the fridge. So I should go to the fridge first. </think> <action> go to fridge 1 </action> H.2.4. SOKOBAN Sokoban Rollout Example (one turn) Input: user You are an expert agent operating in the Sokoban environment. Your goal is to push all the boxes onto the target spots. Once all boxes are on the targets, you win! # Rules You can only push boxes. You can’t pull them, so plan ahead to avoid getting stuck. You can’t walk through or push boxes into walls. To avoid traps, do not push boxes into corners or against walls where they can’t be moved again. # Visual Elements in the Image: Character: A small, green alien-like figure with two antennae and black eyes. It represents you. Box: A yellow crate marked with an orange "X" across its front. Target: A black tile outlined in red, with a small red diamond shape in the center. # Current Step Your current observation is shown in the image. Your admissible actions are ["up", "down ", "left", "right"]. Now it’s your turn to make a move ( choose ONE action only for the current step). You should first reason step-by-step about the current situation - observe the positions of boxes and targets, plan a path to push a box toward a target, and avoid traps like corners or walls. This reasoning process MUST be enclosed within <think> </think> tags. Once you’ve finished your reasoning, you should choose an admissible action for the current step and present it within <action> </action> tags. Output: <think> 26 ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning To push the box to the target, I need to move it to the right. The current position is a corner, which is a trap. I should avoid pushing the box into corners or against walls. The path is straightforward: move right. This action will allow me to push the box towards the target without getting stuck. </think> <action> right </action> 27