Paper deep dive
PlanPO: Group Planning-Aware Policy Optimization for Multi-Turn Agentic LLMs
Dayang Liang, Liyuan He, Xuan Feng, Shuxin Li, Bo An, Yunlong Liu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/19/2026, 4:41:35 AM
Summary
The paper introduces PlanPO (Group Planning-aware Policy Optimization), a reinforcement learning method for training agentic Large Language Models (LLMs) on multi-turn interactive tasks. PlanPO addresses the issue of 'advantage collapse' in existing group-relative methods like GRPO, where successful trajectories with different efficiencies receive identical rewards. PlanPO introduces coarse-to-fine advantage signals by normalizing outcome rewards by trajectory-level and turn-level response lengths, conditioned on success. This encourages agents to learn generalizable planning abilities and efficient behaviors. Experiments show PlanPO outperforms GRPO by 27.2% on average across ALFWorld, WebShop, and SciWorld benchmarks.
Entities (10)
Relation Signals (8)
PlanPO → evaluatedon → WebShop
confidence 95% · We evaluate PlanPO on three challenging multi-turn benchmarks, ALFWorld, WebShop, and SciWorld
PlanPO → evaluatedon → SciWorld
confidence 95% · We evaluate PlanPO on three challenging multi-turn benchmarks, ALFWorld, WebShop, and SciWorld
PlanPO → evaluatedon → ALFWorld
confidence 95% · We evaluate PlanPO on three challenging multi-turn benchmarks, ALFWorld, WebShop, and SciWorld
PlanPO → improvesover → GRPO
confidence 95% · PlanPO improves over GRPO by 27.2% on average across the challenging multi-turn benchmarks
GRPO → isbaselinefor → PlanPO
confidence 90% · PlanPO improves over GRPO by 27.2% on average
PlanPO → uses → coarse-to-fine advantage signals
confidence 90% · PlanPO introduces coarse-to-fine advantage signals, which capture the relative differences in trajectory-level lengths and turn-level response lengths
PlanPO → usesmodel → Qwen2.5-1.5B-Instruct
confidence 90% · using Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct
PlanPO → usesmodel → Qwen2.5-7B-Instruct
confidence 90% · using Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Group-relative policy optimization has emerged as a key paradigm for training agentic large language models (LLMs) on multi-turn interactive tasks. However, most existing variants fail to distinguish advantages among successful trajectories even when these trajectories differ substantially in their interaction efficiency. For instance, circuitous successes are often assigned the identical outcome reward, causing advantage collapse and severe performance bottlenecks. To this end, we propose Group Planning-aware Policy Optimization (PlanPO), a simple yet effective RL method for learning generalizable planning abilities beyond task-specific high-quality behavior patterns. Specifically, PlanPO introduces coarse-to-fine advantage signals, which capture the relative differences in trajectory-level lengths and turn-level response lengths conditioned on successful trajectories sampled for the same task. Within the group-relative optimization structure, this enables agents to actively learn generalizable and deliberate behaviors spanning interaction planning and textual generation from high-quality rollouts, without degenerating into vanilla length minimization. Experimentally, PlanPO improves over GRPO by 27.2\% on average across the challenging multi-turn benchmarks ALFWorld, WebShop, and SciWorld, outperforming recent powerful baselines while incurring negligible additional training cost.
Tags
Links
- Source: https://arxiv.org/abs/2608.17289v1
- Canonical: https://arxiv.org/abs/2608.17289v1
Trouble viewing inline? Open PDF directly →
Full Text
49,860 characters extracted from source content.
Expand or collapse full text
PlanPO: Group Planning-Aware Policy Optimization for Multi-Turn Agentic LLMs Dayang Liang Liyuan He Xuan Feng Shuxin Li Bo An Yunlong Liu Abstract Group-relative policy optimization has emerged as a key paradigm for training agentic large language models (LLMs) on multi-turn interactive tasks. However, most existing variants fail to distinguish advantages among successful trajectories even when these trajectories differ substantially in their interaction efficiency. For instance, circuitous successes are often assigned the identical outcome reward, causing advantage collapse and severe performance bottlenecks. To this end, we propose Group Planning-aware Policy Optimization (PlanPO), a simple yet effective RL method for learning generalizable planning abilities beyond task-specific high-quality behavior patterns. Specifically, PlanPO introduces coarse-to-fine advantage signals, which capture the relative differences in trajectory-level lengths and turn-level response lengths conditioned on successful trajectories sampled for the same task. Within the group-relative optimization structure, this enables agents to actively learn generalizable and deliberate behaviors spanning interaction planning and textual generation from high-quality rollouts, without degenerating into vanilla length minimization. Experimentally, PlanPO improves over GRPO by 27.2% on average across the challenging multi-turn benchmarks ALFWorld, WebShop, and SciWorld, outperforming recent powerful baselines while incurring negligible additional training cost. Introduction Large Language Models (LLMs) have demonstrated remarkable progress across a wide range of complex multi-turn tasks, including information retrieval (17; 8), web navigation (25)(45), code generation (48; 7), and embodied interaction (14; 49). Recent work has increasingly explored agentic reinforcement learning (RL) with verifiable outcome rewards (52; 7; 15), particularly Group Relative Policy Optimization (GRPO) (27), to fine-tune open-source LLMs such as Qwen2.5 (39), thereby improving the capabilities of LLM agents in multi-turn tasks. However, a central challenge in on-policy agentic RL lies in enriching the reward signals of rollout trajectories while improving data utilization (44; 50; 34; 24). Many early studies addressed this issue by introducing value models, such as critics (26; 6) and process reward models (PRMs) (3), to evaluate turn-level behaviors. Yet these models may introduce estimation or proxy bias, and incur substantial memory costs (3). Recent group-optimization approaches instead mine informative rollouts to craft turn-level reward signals and achieve discriminative advantages. For example, HiPER (24) computes the multiple returns of trajectory segments via sub-task decomposition, while GiGPO (9) constructs step advantage signals over the action space by identifying repeated anchor states. Other methods, such as R3L (29) and GVPO (47), establish turn-level credit from failure reflection and diverse code execution feedback. However, most methods rely on labor-intensive manual design and empirical heuristics, which limits their generality across task settings. As a result, advantage collapse within successful rollout groups remains difficult to mitigate in a broadly applicable way (51). This motivates the question of how to enrich rollout-driven training signals for long-horizon tasks in a simple, effective, and more task-general manner. We begin by revisiting a naturally available yet underutilized signal in group rollouts, i.e., the length profiles across both turn-level interaction trajectories and token-level generated responses. Figure 1 provides an abstract illustration of this intuition. Specifically, our key observation is that many inefficiencies in agentic RL manifest as excessive interaction or generation length. In multi-turn interactions, agents may hesitate between states, repeatedly visit similar observations, or enter dead ends before eventually completing the task. A similar issue arises in token-level textual responses. Given the same question or turn-level observation, sampled responses may produce correct actions while still containing unnecessarily verbose, convoluted, or even logically flawed reasoning traces. Nevertheless, both inefficient turns and reasoning tokens can still share identical success rewards just like the superior solutions. Crucially, treating such heterogeneous successes as equally preferable weakens distinguishable signals and the underlying abilities, while allowing noisy rollouts to degrade training quality and impose substantial performance bottlenecks. To address this, we present Group Planning-aware Policy Optimization (PlanPO), an effective group-based RL method for learning generalizable planning abilities beyond specific planful behaviors. Specifically, within the set of successful trajectories sampled in same task, PlanPO normalizes outcome rewards by turn-level trajectory lengths and token-level response lengths, constructing coarse-to-fine dense reward signals. Rather than directly summing these dense rewards, PlanPO then computes their relative advantages separately and combines them through a weighted formulation, preserving discriminative information at different granularities (20). Additionally, unlike generic length-based reward shaping (23; 21), PlanPO performs multiscale length normalization only conditioned on successes with group-relative structure. Thus, successful completion is a prerequisite for group length normalization. Empirically, we show that these success-conditioned relative advantages help agents acquire planning behaviors that generalize beyond specific high-quality trajectories, rather than merely imitating task-specific success patterns. Figure 1: Left: Successful rollouts are not equally informative. Rollouts that reach the same task goal, can differ substantially in length, directness, and reasoning quality, while the most optimal strategies may reveal more sophisticated and generalizable capabilities. Right: Averaged normalization performance comparison across ALFWorld, WebShop, and SciWorld environments using the Qwen2.5-1.5b model. We evaluate PlanPO on three challenging multi-turn benchmarks, ALFWorld (31), WebShop (40), and SciWorld (36), using Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct. PlanPO consistently outperforms recent strong baselines while incurring negligible additional training cost, specifically improving over GRPO by 27.2% on average and notably achieving a 24.3% gain on out-of-distribution tasks in ALFWorld. Extensive ablations and analyses further validate its effectiveness and generalizability. Our contributions are threefold. • We identify successful-rollout heterogeneity as a critical bottleneck in group-relative optimization, where redundant turns and reasoning can mask rollout quality and then limit policy learning. • We propose PlanPO, which constructs success-conditioned length-normalized advantages, encouraging agents to learning planning-aware abilities beyond specific behavioral patterns. • We show that PlanPO consistently outperforms recent strong baselines across ALFWorld, WebShop, and SciWorld, while incurring negligible additional training cost. Preliminaries Multi-Turn Agentic RL. We consider RL for LLM-based agents that interact with an environment over multiple turns, where the interaction process is formulated as a finite-horizon Markov Decision Process (MDP). Given a task instance ∈p(X) x∈ p(X), at each turn t=1,2,…,Tt=1,2,…,T, an agent policy πθ _θ observes a state t∈ s_t and generates a textual action t∈ a_t , and then transitions to the next state t+1∈ s_t+1 while yielding a scalar reward rt∈ℝr_t . The interaction unfolds as a trajectory =(1,1,r1),(2,2,r2),…,(T,T,rT) τ=\( s_1, a_1,r_1),( s_2, a_2,r_2),...,( s_T, a_T,r_T)\, where T denotes the trajectory length in interaction turns. In practical tasks such as ALFWorld, the agent generates a response as its action for each observation. This response is structured within <think></think> and <action></action> tags, where the former contains the reasoning process, while the latter specifies the actual action executed in the environment. Notably, for most task settings, reward signals are sparse and delayed, e.g., the environment provides an outcome reward R()R( τ) only after the trajectory terminates. Group-relative Policy Optimization Recent agentic RL methods for LLMs commonly adopt a group-relative policy optimization paradigm. Given a task instance x, the old policy πθold _ _ old samples a group of N candidate trajectories x=1,2,…,NG_x=\ τ_1, τ_2,…, τ_N\, where each trajectory corresponds to one complete rollout. Each trajectory i τ_i receives a scalar reward R(i)R( τ_i) that reflects the overall quality or success of the generated outcome. Instead of learning an advantage function A(t,t)A( s_t, a_t) with critic networks in PPO (26), group-based RL computes the advantage using only statistics within the sampled group: A(i)=(R(i)i=1N).A( τ_i)= GroupNormalization (\R( τ_i)\_i=1^N ). In GRPO (27), the advantage is evaluated by normalizing each trajectory reward with the mean and variance of group rewards (R(i)i=1N)(\R( τ_i)\_i=1^N). This sampling-based estimator reduces the memory and computational overhead introduced by the critic architecture in conventional PPO. Figure 2: Overview of PlanPO. Left: Rollouts sampled for the same task can reach the same outcome through interaction paths and textual responses of markedly different quality. Middle: PlanPO converts the outcome into success-conditioned trajectory- and turn-level scores using trajectory and response lengths, respectively. Right: The two scores are normalized separately within the same-task group and combined into a coarse-to-fine advantage for policy optimization. Group Planning-Aware Policy Optimization for Multi-turn Agentic LLMs We propose group-relative Planning-aware Policy Optimization (PlanPO), a simple and effective group-relative RL method for learning high-level planning strategies beyond task-specific high-quality rollouts. We begin with our motivation, followed by the introduction of the coarse-to-fine advantage design, and conclude with the policy optimization loss and theoretical analysis. Motivation As illustrated in the right of Figure 2, given the same task, some rollouts reach the goal through short and coherent interaction paths, whereas others involve redundant state transitions or even enter loops and dead ends. At the turn level, verbose reasoning can also introduce inconsistencies or misleading intermediate claims. For example, as illustrated, given a task or turn observation, although the final sampled actions "<action>take potato 1 from countertop 1</action>" are all correct, the thought process "I have already taken the potato ..." treats it as already happened. Such cases are common in the rollouts yet introduce hallucinated or logically inconsistent reasoning traces. How should we quantify the quality of successful rollouts? Existing solutions may introduce post-hoc reflection (29) or agentic verifier models (46), but these signals are often costly and heuristic. The aforementioned observations motivate us to leverage multiscale rollout length as an initial signal. However, unconditional length-based reward shaping would degenerate into meaningless length minimization, even distorting the representation space of LLMs. Our goal is instead to comparatively learn planful behaviors within successful rollouts, while promoting task-general planning capabilities rather than fitting task-specific patterns. This objective naturally aligns with group-relative policy optimization. Below, we progressively construct the above conditional advantage signals. Trajectory Length-Normalized Advantage To achieve this, we first instantiate the success-conditioned length signal at the trajectory (or episode) level. For successful rollouts with the same outcome reward, trajectory length serves as a coarse proxy for planning efficiency across environment interactions. We therefore normalize the outcome reward by trajectory length only for successful rollouts, and compute the conditional group-relative advantage. Formally, given a task instance x, the policy πθold _ _ old samples N trajectories 1,2,…,N\ τ_1, τ_2,…, τ_N\ start from the identical and initial state 0 s_0. Each rollout trajectory is represented as i=(i,1,i,1,ri,1),(i,2,i,2,ri,2),…,(i,Ti,i,Ti,ri,Ti) τ_i=\( s_i,1, a_i,1,r_i,1),( s_i,2, a_i,2,r_i,2),...,( s_i,T_i, a_i,T_i,r_i,T_i)\, where TiT_i represents the number of turns in i-th trajectory. In our task setting, each trajectory receives only a terminal outcome reward R(i)=10R( τ_i)=10 when the task goal is reached. We then denote the trajectory-level group of sampled trajectories and rewards as: E=(1,R(1)),(2,R(2)),…,(N,R(N)),G_ x^E= \ ( τ_1,R( τ_1) ), ( τ_2,R( τ_2) ),…, ( τ_N,R( τ_N) ) \, (1) where the superscript EE denotes the episode, i.e., trajectory level. Let U⊆EG_ x^U _ x^E denote the subset of successful trajectories, and let [i∈U] 1[ τ_i _ x^U] indicate whether i τ_i succeeds. For each trajectory, we define its length-normalized episode reward as, RE(i)=[i∈U]R(i)/Ti,R^E( τ_i)= 1[ τ_i _ x^U]\,R( τ_i)/T_i, (2) where TiT_i is the number of valid interaction turns in i τ_i. The corresponding trajectory-level advantage AEA^E is computed by group-relative normalization: AE(i)=RE(i)−mean(RE(j)j=1N)Fnorm(RE(j)j=1N).A^E( τ_i)= R^E( τ_i)-mean ( \R^E( τ_j) \_j=1^N )F_norm ( \R^E( τ_j) \_j=1^N ). (3) Here, Fnorm(⋅)F_norm(·) is the normalization factor, instantiated as either std(⋅)+ϵstd(·)+ε or 11 (9), with the former usually used by default, where ϵε is a small constant for numerical stability (27). The trajectory-level AE(i)A^E( τ_i) is broadcast to each turn in the i-th trajectory, providing a trajectory-wide discrimination signal among successful rollout trajectories that complete the same task but differ in process quality or long-term planning. Response Length-Normalized Advantage While the above advantage provides learning signals for discriminating trajectories, each response turn within a trajectory still lacks fine-grained credit. Next, we compute group-relative advantages only among successful responses, so that the agent learns that planful responses are desirable only when the corresponding actions remain correct. Additionally, we flatten all successful responses generated for the same task into one group for advantage computation. We then employ a similar group relative advantage structure to achieve the above idea. Formally, for the t-th interaction turn in trajectory i τ_i, we collect all active responses into a turn-level group, ,tS=(i,t,R(i))∣i∈E,t≤Ti,G_ x,t^S= \ ( a_i,t,R( τ_i) ) τ_i _ x^E,\ t≤ T_i \, (4) where SS denotes the step, i.e., turn level, and Li,t=|i,t|L_i,t=| a_i,t| is the token length of the response generated by trajectory i τ_i at turn t. Similar to the trajectory-level case, we define the response length-normalized reward as, RS(i,t)=[i∈U]R(i)/Li,t.R^S( a_i,t)= 1[ τ_i _ x^U]R( τ_i)/L_i,t. (5) The turn-level relative advantage is then computed within the active response group: AS(i,t)=RS(i,t)−mean(RS(j,t)∣j,t∈,tS)Fnorm(RS(j,t)∣j,t∈,tS).A^S( a_i,t)= R^S( a_i,t)-mean ( \R^S( a_j,t) a_j,t _ x,t^S \ )F_norm ( \R^S( a_j,t) a_j,t _ x,t^S \ ). (6) Here, the resulting turn advantage AS(i,t)A^S( a_i,t) is assigned to all tokens in the response i,t a_i,t, which provides a fine-grained supervision signal for successful responses. Coarse-to-Fine Group Policy Optimization We integrate the trajectory-level and turn-level advantages into a multiscale group-relative advantage for policy optimization. For the response i,t a_i,t in trajectory i τ_i, the final PlanPO advantage is defined as, APlanPO(i,t)=AE(i)+α(k)AS(i,t),A^PlanPO( a_i,t)=A^E( τ_i)+α(k)A^S( a_i,t), (7) withα(k)=(k,αinit,αfinal),with~~α(k)= LinearDecay(k; _ init, _ final), (8) where α(k)α(k) denotes the decay weight with training step k for balancing the two level signals. We set the turn-level coefficient smaller than the trajectory-level coefficient, i.e., 0≤αfinal≤α(k)<αinit≤10≤ _ final≤α(k)< _ init≤ 1. This design keeps trajectory-level planning quality as the dominant signal and uses response length signals only as a refinement. Although the response constraint is applied only to successful trajectories, an overly large α(k)α(k) experimentally over-penalize response length, thereby hurting task performance. Please see the ablation study for details. Therefore, we gradually decay α(k)α(k) during training to reduce the strength of response-length normalization as the policy becomes more capable. Finally, PlanPO optimizes the policy with the same clipped group-relative objective backbone as GRPO-style methods: PlanPO(θ) ^PlanPO(θ) =∼p(X),ii=1N∼πθold[1∑i=1NTi∑i=1N∑t=1Tiℓi,t =E_ subarrayc x p(X),\\ \ τ_i\_i=1^N _ _ old subarray [ 1 _i=1^NT_i _i=1^N _t=1^T_i _i,t (9) −βKL(πθ(⋅∣)∥πref(⋅∣))], - _KL ( _θ(· x)\| _ ref(· x) ) ], ℓi,t=min(CLOSE _i,t= ( ρθ(i,t)APlanPO(i,t), _θ( a_i,t)A^PlanPO( a_i,t), (10) OPENclip(ρθ(i,t),1−ϵ,1+ϵ)APlanPO(i,t)). ( _θ( a_i,t),1-ε,1+ε)A^PlanPO( a_i,t) ). Here, ρθ(i,t)=πθ(i,t∣i,t,)/πθold(i,t∣i,t,) _θ( a_i,t)= _θ( a_i,t s_i,t, x)/ _ _ old( a_i,t s_i,t, x) is the importance sampling ratio, ϵε is the clipping coefficient, and β controls the strength of KL regularization with respect to the reference policy πref _ ref. Type Method ALFWorld WebShop Pick Look Clean Heat Cool Pick2 All Score Succ. Closed-Source Model Prompting GPT-4o 75.3 60.8 31.2 56.7 21.6 49.8 48.0 31.8 23.7 Prompting Gemini-2.5-Pro 92.8 63.3 62.1 69.0 26.6 58.7 60.3 42.5 35.9 Qwen2.5-1.5B-Instruct Prompting ReAct 17.4 20.5 15.7 6.2 7.7 2.0 12.8 40.1 11.3 Prompting Reflexion 35.3 22.2 21.7 13.6 19.4 3.7 21.8 55.8 21.9 RL Training PPO (with critic) 64.8±3.5 40.5±6.9 57.1±4.9 60.6±6.6 46.4±4.0 47.4±1.9 54.4±3.1 73.8±3.0 51.5±2.9 RL Training RLOO 88.3±3.0 52.8±8.6 71.0±5.9 62.8±8.7 66.4±5.5 56.9±4.7 69.7±2.5 73.9±5.6 52.1±6.7 RL Training GRPO 85.3±1.5 53.7±8.0 84.5±6.8 78.2±7.9 59.7±5.0 53.5±5.6 72.8±3.6 75.8±3.5 56.8±3.8 RL Training EMPG 85.5 33.5 78.9 76.2 74.7 89.1 73.7 80.4 60.8 RL Training GiGPOw/ std 94.4±5.9 67.5±4.6 94.8±3.8 94.4±7.8 79.8±4.7 76.4±5.4 86.7±1.7 83.1±1.6 65.0±3.2 RL Training GiGPOw/o std 96.0±1.4 76.5±3.9 91.8±5.5 91.3±6.3 71.7±8.4 79.5±7.7 86.1±4.7 83.5±1.8 67.4±4.5 RL Training PlanPO 98.2±1.1 85.1±4.6 94.6±4.7 93.8±6.0 82.4±6.7 83.7±8.0 91.3±4.1 86.8±1.5 77.2±5.6 Qwen2.5-7B-Instruct Prompting ReAct 48.5 35.4 34.3 13.2 18.2 17.6 31.2 46.2 19.5 Prompting Reflexion 62.0 41.6 44.9 30.9 36.3 23.8 42.7 58.1 28.8 RL Training PPO (with critic) 92.3±4.0 64.0±8.4 92.5±2.4 89.5±7.0 80.3±2.0 68.8±8.3 80.4±2.7 81.4±3.1 68.7±5.1 RL Training RLOO 87.6±4.3 78.2±8.3 87.3±5.8 81.3±7.6 71.9±5.2 48.9±8.4 75.5±4.6 80.3±3.2 65.7±4.0 RL Training GRPO 90.8±5.1 66.1±6.7 89.3±5.4 74.7±6.9 72.5±5.4 64.7±7.3 77.6±5.2 79.3±2.8 66.1±3.7 RL Training EMPG 92.9 75.2 74.8 86.3 73.7 65.3 78.5 81.0 69.3 RL Training GiGPOw/ std 97.7±1.6 82.7±7.9 98.8±1.6 83.7±7.2 89.3±8.2 79.2±6.6 90.8±1.3 84.4±2.9 72.8±3.2 RL Training GiGPOw/o std 91.8±5.4 88.6±6.3 95.9±3.2 90.2±2.6 86.5±5.5 85.2±7.5 90.2±2.3 86.2±2.6 75.2±3.8 RL Training PlanPO 100.0±0.0 92.1±6.3 96.8±2.5 99.0±1.4 88.3±6.8 89.7±7.8 94.4±2.1 88.5±3.2 80.6±4.7 Table 1: Evaluation results on ALFWorld and WebShop. For each RL training method, we report the mean and variance over three random seeds. The ALFWorld contains six categories: Pick & Place (Pick), Examine in Light (Look), Clean & Place (Clean), Heat & Place (Heat), Cool & Place (Cool), and Pick Two & Place (Pick2). Most entries in this table are reported by Feng et al. (9). Notably, the baseline GiGPOw/ std denotes Fnorm=stdF_norm=std, and GiGPOw/o std denotes Fnorm=1F_norm=1. Type Model Measure Test-Cond. Find Chem-Mix Lifespan Overall Proprietary GPT-4o 15.09 6.02 38.64 20.00 73.33 21.00 Qwen-Max 9.43 0.00 34.09 20.00 40.00 13.50 Gemini-2.5-Flash 11.32 0.00 54.55 0.00 80.00 21.00 OpenAI o4-mini 20.75 14.46 47.73 0.00 100.00 29.50 OpenAI o3 47.17 25.30 56.82 40.00 66.67 41.50 Gemini-2.5-Pro 9.43 0.00 29.55 0.00 46.67 12.50 Models ≥ 100B Qwen3-235B-A22B 11.32 4.82 59.09 20.00 66.67 23.50 DeepSeek-V3-0324 0.00 0.00 2.27 0.00 0.00 0.50 DeepSeek-R1-0528 1.89 0.00 11.36 0.00 20.00 4.50 Models <<100B Qwen2.5-72B-Instruct 7.55 1.20 15.91 20.00 40.00 9.50 Qwen3-32B 5.66 1.20 31.82 0.00 66.67 14.00 Llama-3.1-70B-Instruct 24.53 4.82 40.91 40.00 86.67 25.00 RL Training AgentGym-RL-3B 20.75 28.92 0.00 0.00 66.67 22.50 AgentGym-RL-7B 24.53 59.04 65.91 0.00 66.67 50.50 ScalingInter-7B 33.96 55.42 88.64 0.00 73.33 57.00 PlanPO-7B 40.52 69.38 91.67 0.00 85.94 68.46 Table 2: Evaluation results on SciWorld over three random seeds. The task Test-Cond. means test conductivity and Chem-Mix means chemistry mix. Baseline entries are reported by Zhang et al. (46). Figure 3: Left two panels: Ablation study on the weight schedule α(k)α(k) with αinit=2αfinal _init=2 _final in PlanPO. Third panel: Component ablation. We compared the ablation results of PlanPO with default decay, trajectory-level advantage AEA^E only, turn-level advantage ASA^S only, and PlanPO with constant α=0.1α=0.1. Right panel: Runtime breakdown of each PlanPO iteration on four NVIDIA A40 GPUs. Compared to the GRPO, the additional computation introduced by PlanPO lies in the reward and advantage stages, which is negligible. Conversely, PlanPO can reduce overall task runtime by 12.5% Theorem 1 (Bias–variance trade-off in PlanPO). Consider a fixed response ai,ta_i,t at optimization step k, with the randomness induced by repeated same-task group sampling. Let AE,⋆=[AE]A^E, =E_G[A^E] and AS,⋆=[AS]A^S, =E_G[A^S] denote the expected trajectory-level signal and turn-level refinement, respectively. Assume that their centered estimation errors have variances vEv_E and vSv_S and non-negative covariance c. Relative to the full-refinement target A⋆=AE,⋆+AS,⋆A =A^E, +A^S, , the PlanPO estimator Aα=AE+αASA_α=A^E+α A^S, where α∈[0,1]α∈[0,1], satisfies Bias2(Aα) ^2(A_α) =(1−α)2(AS,⋆)2, =(1-α)^2(A^S, )^2, Var(Aα) (A_α) =vE+α2vS+2αc. =v_E+α^2v_S+2α c. Consequently, increasing α decreases the squared bias while increasing the variance, with: 0 0 ≤Bias2(Aα)≤(AS,⋆)2, ^2(A_α)≤(A^S, )^2, vE v_E ≤Var(Aα)≤(vE+vS)2. (A_α)≤ ( v_E+ v_S )^2. This theorem shows that α(k)α(k) relates to PlanPO’s bias–variance trade-off. A larger α(k)α(k) preserves more turn-level refinement but raises sampling variance, whereas a smaller α(k)α(k) reduces variance at the cost of greater shrinkage bias. The technical supplement provides detailed proof. Experiments We evaluate PlanPO across a range of multi-turn environments. Specifically, the experiments aim to answer three questions: (1) How well does PlanPO perform overall across these challenging environments? (2) How does PlanPO achieve its performance gains? (3) Does PlanPO acquire the capabilities of task generalization and planning awareness rather than overfitting to task-specific success patterns? Experimental Setup Benchmarks. We train and evaluate PlanPO on three challenging multi-turn benchmarks: ALFWorld (31), WebShop (40), and SciWorld (36). ALFWorld is an embodied household environment for evaluating long-horizon textual reasoning and decision-making. In each episode, the agent receives a concrete task instruction sampled from 3,8273,827 tasks from six categories. WebShop is an interactive web-shopping environment containing nearly 1.11.1 million products and 12,00012,000 user instructions. SciWorld is designed for scientific tasks and provides APIs through which agents can manipulate scientific instruments and conduct experiments. The technical supplement provides further descriptions of the benchmarks. Baselines. For ALFWorld and WebShop, we compare PlanPO against three categories of competitive baselines: (1) Proprietary models: GPT-4o (1) and Gemini-2.5-Pro (32); (2) training-free prompting agents: ReAct (41) and Reflexion (30); and (3) RL-based methods: PPO (26), RLOO (2), GRPO (27), EMPG (35), and GiGPO (9), which cover representative actor–critic and group-relative optimization approaches. For SciWorld, we consider (1) training-free agents based on the OpenAI (1), Gemini (32), Qwen (38), Llama (10), and DeepSeek (19; 11) model families, spanning both proprietary models and open-source models of different scales; and (2) RL-based methods: AgentGym-RL (37), a GRPO framework for training across multiple environments, and ScalingInter (37), a group-based RL method that reports strong performance on SciWorld. Implementation details. We employ Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct as the base models for training experiments. To ensure fair comparisons, the hyperparameter settings of PlanPO follow the existing RL framework (9; 37) in each benchmark. Specifically, we use a group size of N=8N=8, a learning rate of 1×10−61× 10^-6, and a KL-penalty coefficient of 1×10−31× 10^-3 for SciWorld and 0.010.01 for the other environments. The maximum number of interaction turns is set to 5050 for ALFWorld, 1515 for WebShop, and 2020 for SciWorld. We train for 150150 steps on ALFWorld and WebShop and for 200200 steps on SciWorld. On ALFWorld and WebShop, we set the turn-level weight αinit=2αfinal=0.1 _init=2 _final=0.1 in PlanPO. For the exploratory SciWorld, we reduce it to 0.050.05 without further tuning to improve training stability. Finally, all our experiments were run on 6 NVIDIA H200s and 8 NVIDIA A40s. The technical supplement provides additional implementation details. Experiment Results Performance on ALFWorld and WebShop. As shown in Table 1, PlanPO substantially outperforms advanced closed-source models on both benchmarks. For the open-source Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct models, RL methods are significantly stronger than prompting-based methods. Nevertheless, PlanPO also outperforms existing powerful baselines, e.g., GiGPO and EMPG. For example, on the 1.5B model, PlanPO achieves an overall success rate of 91.3%91.3\% across the six ALFWorld categories, improving over GRPO by +18.5+18.5 points, while requiring only several regularization operations with negligible computational cost. Performance on SciWorld. In Table 2, PlanPO achieves consistently the best overall scores on complex and exploratory scientific scenarios, notably improving the GRPO-based AgentGym-RL from 50.550.5 to 68.4668.46. However, we observe that RL training methods exhibit a consistent failure pattern in the Chem-Mix domain, which may be due to the model’s limited understanding and exploration ability required for rigorous scientific analysis (37). In-depth Analysis Ablation Study. To verify how PlanPO achieves performance gains, we performed two ablation studies on the scheduling coefficient α(k)α(k) and each component. As shown in left Figure 3, we compared a series of initial αinit=0.5,0.2,0.1,0.05,0 _init=\0.5,0.2,0.1,0.05,0\. We observed that αinit=0.1 _init=0.1 achieved the best performance, while both smaller values (e.g., αinit=0 _init=0) and excessively large values (e.g., 0.50.5) degraded performance. This matches the theoretical bias–variance trade-off, i.e., the turn-level signal is useful as a refinement, but should not dominate the trajectory-level planning signal. In the third panel of Figure 3, we observe that ablating any level signal significantly degrades PlanPO’s performance, especially without AEA^E. Regarding turn-level advantages, PlanPO with coefficient decay settings consistently outperforms those using constant coefficients. In summary, its performance gains primarily stem from the AEA^E, while a properly weighted ASA^S further enables optimal performance. Type Method In-Success Out-Success Prompting GPT-4o 48.0 46.0 Prompting Gemini2.5 60.3 50.5 RL Training PPO 54.4±3.1 50.9±7.6 RL Training RLOO 69.7±2.5 68.7±10.7 RL Training GRPO 72.8±3.6 70.1±2.5 RL Training GiGPO 86.7±1.7 82.4±2.0 RL Training PlanPO 91.3±4.1 87.1±3.6 Table 3: Generalization evaluation with three seeds in ALFWorld using Qwen2.5-1.5B-Instruct. Some entries are reported by He et al. (12). The In-Success reports the success rate on the in-distribution tasks, while the Out-Success reports the success rate on out-of-distribution tasks. Generalization Verification. The gains of PlanPO remain strong when the evaluation tasks differ from the training distribution. As shown in Table 3, PlanPO achieves 87.1%87.1\% success on out-of-distribution ALFWorld tasks, improving over GRPO by +17.0+17.0 points and over GiGPO by +4.7+4.7 points. The gap between in-distribution and out-of-distribution performance is also modest, decreasing from 91.3%91.3\% to 87.1%87.1\%. This suggests that group-based PlanPO does not merely memorize some better planning templates, but learns planning-aware abilities that generalize to unseen task configurations. Figure 4: Length-normalized Comparison Analysis with different reward shaping settings in ALFWorld. Figure 5: Mean Length Comparison in ALFWorld tasks. Length-Normalized Method Comparison. Figure 4 further isolates the role of success-conditioned length normalization. Directly applying unconditional length shaping in PPO quickly collapses to nearly zero success, showing that shorter generations or trajectories are not intrinsically better. Success conditioning alone is also insufficient, as PPO with success-conditioned length shaping remains close to standard PPO. By contrast, PlanPO steadily improves after training begins and reaches around 0.550.55 success near the end, clearly above GRPO and PPO variants. These results indicate that the key benefit comes from coupling length-induced signals with group-relative policy optimization, which turns heterogeneous successful rollouts into useful supervision. Note that the reward shaping operation and outcome reward size R(τ)=10R(τ)=10 involved are consistent with PlanPO settings, except for the condition control and basic algorithm. Planning-Aware Strategies. We further examine whether PlanPO induces planning-aware behavior beyond improving final task success. As shown in Figure 5, PlanPO consistently produces shorter trajectories than GRPO during training, reducing the mean number of interaction turns to 13.813.8 compared with 26.126.1 for GRPO. This suggests that PlanPO learns to reach task goals through more direct interaction paths, avoiding redundant transitions, repeated trials, and unnecessary detours. At the response level, PlanPO also yields more compact generations, with the mean response length decreasing to 56.356.3 tokens, while GRPO remains substantially more verbose at 95.195.1 tokens. Importantly, this behavior does not arise from naive length minimization: as shown in Figure 4, unconditional length shaping hurts task success, whereas PlanPO applies length normalization only under success-conditioned group-relative comparisons. These results indicate that PlanPO encourages planning-aware strategies by preferring successful rollouts that are both interaction-efficient and response-concise, thereby turning heterogeneity among successful trajectories into a useful signal for learning generalizable agentic planning abilities. Related Work RL has become a central recipe for improving LLM reasoning and interactive agents (28; 27; 43; 33). PPO-based RLHF (26) trains a value model to reduce variance, while lighter critic-free alternatives such as ReMax (16), RLOO (2), and GRPO (27) replace the learned critic with sampled baselines or group-relative rewards, improving scalability for LLMs. Recent GRPO variants further refine this objective (21; 5; 54; 18). DAPO improves stability with decoupled clipping, dynamic sampling, and overlong-response shaping (42), Dr. GRPO analyzes length and reward-variance normalization biases (22), and GMPO stabilizes updates by changing the aggregation of token-level rewards (53). In parallel, agentic RL methods extend outcome-reward optimization to long-horizon interaction, including WebSailor (13), GiGPO (9), HGPO (12), and A2TGPO (4), which study credit assignment through episode-, state-, history-, or turn-grouped advantages. Different from these methods, our work focuses on planning-aware credit assignment under successful-rollout heterogeneity. Conclusion This paper addresses successful-rollout heterogeneity in group-relative RL for multi-turn agentic LLMs. We propose PlanPO, which constructs success-conditioned length-induced advantages from trajectory-level interaction and token-level response generation. The experiments across multiple benchmarks demonstrate that PlanPO consistently outperforms existing methods and achieves the best task performance with stronger generalization, planful Strategies, and negligible overhead. Our results suggest that successful-rollout heterogeneity provides a simple and scalable signal for learning planning-aware and generalizable behaviors. References Achiam et al. (2023) J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. Cited by: Baselines.. Ahmadian et al. (2024) A. Ahmadian, C. Cremer, M. Gallé, M. Fadaee, J. Kreutzer, O. Pietquin, A. Üstün, and S. Hooker Back to basics: revisiting reinforce-style optimization for learning from human feedback in llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 12248–12267. Cited by: Baselines., Related Work. Chae et al. (2026) H. Chae, S. Kim, J. Cho, S. Kim, S. Moon, G. Hwangbo, D. Lim, M. Kim, Y. Hwang, M. Gwak, et al. Web-shepherd: advancing prms for reinforcing web agents. Vol. 38. Cited by: Introduction. Chen et al. (2026) D. Chen, Z. Zong, Z. Ma, L. Luo, Y. Li, C. Li, P. Chen, and J. Jiang A2TGPO: agentic turn-group policy optimization with adaptive turn-level clipping. Cited by: Related Work. Cui et al. (2025) G. Cui, Y. Zhang, J. Chen, L. Yuan, Z. Wang, Y. Zuo, H. Li, Y. Fan, H. Chen, W. Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models. Cited by: Related Work. Dai et al. (2025) R. Dai, L. Song, H. Liu, Z. Liang, D. Yu, H. Mi, Z. Tu, R. Liu, T. Zheng, H. Zhu, et al. Cde: curiosity-driven exploration for efficient reinforcement learning in large language models. Cited by: Introduction. Dai et al. (2026) S. Dai, C. Sun, H. Wu, H. Zheng, T. Ji, J. Yan, Y. Wu, D. Zhang, X. Wang, and X. Li Group verification-based policy optimization for interactive coding agents. In The Fourteenth International Conference on Learning Representations, Cited by: Introduction. Eugene et al. (2026) J. Y. Eugene, X. Zhang, Y. Xia, T. Ge, X. Wang, F. Kartik, V. Suryanarayanan, C. Yang, A. Jiang, J. Ding, et al. FormAct: agentic source editing for rich-format document generation. In Forty-third International Conference on Machine Learning, Cited by: Introduction. Feng et al. (2026) L. Feng, Z. Xue, T. Liu, and B. An Group-in-group policy optimization for llm agent training. Vol. 38. Cited by: Introduction, Trajectory Length-Normalized Advantage, Table 1, Baselines., Implementation details., Related Work. Grattafiori et al. (2024) A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The llama 3 herd of models. Cited by: Baselines.. Guo et al. (2025) D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, et al. Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning. Cited by: Baselines.. He et al. (2026) S. He, L. Feng, Q. Wei, X. Cheng, L. Feng, and B. An Hierarchy-of-groups policy optimization for long-horizon agentic tasks. Cited by: Table 3, Related Work. Li et al. (2025) K. Li, Z. Zhang, H. Yin, L. Zhang, L. Ou, J. Wu, W. Yin, B. Li, Z. Tao, X. Wang, et al. Websailor: navigating super-human reasoning for web agent. Cited by: Related Work. Li et al. (2024) M. Li, S. Zhao, Q. Wang, K. Wang, Y. Zhou, S. Srivastava, C. Gokmen, T. Lee, L. E. Li, R. Zhang, et al. Embodied agent interface: benchmarking llms for embodied decision making. Vol. 37. Cited by: Introduction. Li et al. (2026) Y. Li, C. Zhang, R. Lv, A. Liu, K. Deng, Y. Zhang, J. Liu, and B. Zhou Relook: vision-grounded rl with a multimodal llm critic for agentic web coding. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 25471–25485. Cited by: Introduction. Li et al. (2023) Z. Li, T. Xu, Y. Zhang, Z. Lin, Y. Yu, R. Sun, and Z. Luo Remax: a simple, effective, and efficient reinforcement learning method for aligning large language models. Cited by: Related Work. Liao et al. (2025) C. C. Liao, D. Liao, and S. S. Gadiraju Agentmaster: a multi-agent conversational framework using a2a and mcp protocols for multimodal information retrieval and analysis. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, p. 52–72. Cited by: Introduction. Lin et al. (2026) Z. Lin, M. Lin, Y. Xie, and R. Ji Cppo: accelerating the training of group relative policy optimization-based reasoning models. Advances in Neural Information Processing Systems 38, p. 61043–61068. Cited by: Related Work. Liu et al. (2024) A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, et al. Deepseek-v3 technical report. Cited by: Baselines.. Liu et al. (2026) S. Liu, X. Dong, X. Lu, S. Diao, P. Belcak, M. Liu, M. Chen, H. Yin, Y. F. Wang, K. Cheng, et al. Gdpo: group reward-decoupled normalization policy optimization for multi-reward rl optimization. Cited by: Introduction. Liu et al. (2025a) W. Liu, R. Zhou, Y. Deng, Y. Huang, J. Liu, Y. Deng, Y. Zhang, and J. He Learn to reason efficiently with adaptive length-based reward shaping. Cited by: Introduction, Related Work. Liu et al. (2025b) Z. Liu, C. Chen, W. Li, P. Qi, T. Pang, C. Du, W. S. Lee, and M. Lin Understanding r1-zero-like training: a critical perspective. Cited by: Related Work. Pardo et al. (2018) F. Pardo, A. Tavakoli, V. Levdik, and P. Kormushev Time limits in reinforcement learning. In International Conference on Machine Learning, p. 4045–4054. Cited by: Introduction. Peng et al. (2026) J. Peng, Y. Liu, R. Zhou, C. Fleming, Z. Wang, A. Garcia, and M. Hong HiPER: hierarchical plan–execute rl for multi-turn llm agents. In Forty-third International Conference on Machine Learning, Cited by: Introduction. Rawles et al. (2025) C. Rawles, S. Clinckemaillie, Y. Chang, J. Waltz, G. Lau, M. Fair, A. Li, W. Bishop, W. Li, F. Campbell-Ajala, et al. Androidworld: a dynamic benchmarking environment for autonomous agents. In International Conference on Learning Representations, Vol. 2025, p. 406–441. Cited by: Introduction. Schulman et al. (2017) J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov Proximal policy optimization algorithms. Cited by: Introduction, Group-relative Policy Optimization, Baselines., Related Work. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. Deepseekmath: pushing the limits of mathematical reasoning in open language models. Cited by: Introduction, Group-relative Policy Optimization, Trajectory Length-Normalized Advantage, Baselines., Related Work. Sheng et al. (2025) G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu Hybridflow: a flexible and efficient rlhf framework. In Proceedings of the Twentieth European Conference on Computer Systems, p. 1279–1297. Cited by: Related Work. Shi et al. (2026) W. Shi, Y. Chen, Z. Li, X. Pan, Y. Sun, J. Xu, X. Zhou, and Y. Li R3L: reflect-then-retry reinforcement learning with language-guided exploration, pivotal credit, and positive amplification. Cited by: Introduction, Motivation. Shinn et al. (2023) N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao Reflexion: language agents with verbal reinforcement learning. Vol. 36. Cited by: Baselines.. Shridhar et al. (2020) M. Shridhar, X. Yuan, M. Côté, Y. Bisk, A. Trischler, and M. Hausknecht Alfworld: aligning text and embodied environments for interactive learning. Cited by: Introduction, Benchmarks.. Team et al. (2023) G. Team, R. Anil, S. Borgeaud, J. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al. Gemini: a family of highly capable multimodal models. Cited by: Baselines.. Team et al. (2026) K. Team, T. Bai, Y. Bai, Y. Bao, M. C., J. Cai, X. Cai, P. Cao, Y. Cao, Z. Chai, Y. Charles, H. S. Che, G. Chen, G. Chen, G. Chen, et al. Kimi k3: open frontier intelligence. External Links: 2607.24653 Cited by: Related Work. Wang et al. (2026a) G. Wang, S. Dai, G. Ye, Z. Gan, W. Yao, Y. Deng, X. Wu, and Z. Ying Information gain-based policy optimization: a simple and effective approach for multi-turn search agents. In The Fourteenth International Conference on Learning Representations, Cited by: Introduction. Wang et al. (2026b) J. Wang, J. Liu, Y. Fu, Y. Li, X. Wang, Y. Lin, Y. Yue, L. Zhang, Y. Wang, and W. KE Harnessing uncertainty: entropy-modulated policy gradients for long-horizon LLM agents. In Forty-third International Conference on Machine Learning, Cited by: Baselines.. Wang et al. (2022) R. Wang, P. Jansen, M. Côté, and P. Ammanabrolu Scienceworld: is your agent smarter than a 5th grader?. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, p. 11279–11298. Cited by: Introduction, Benchmarks.. Xi et al. (2026) Z. Xi, J. Huang, C. Liao, B. Huang, J. Liu, H. Guo, yajie yang, R. Zheng, J. Ye, J. Zhang, W. Chen, W. He, Y. Ding, G. Li, Z. Chen, Z. Du, X. Yao, Y. Xu, J. Chen, T. Gui, Z. Wu, Q. Zhang, X. Huang, and Y. Jiang AgentGym-rl: an open-source framework to train LLM agents for long-horizon decision making via multi-turn RL. In The Fourteenth International Conference on Learning Representations, Cited by: Baselines., Implementation details., Performance on SciWorld.. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report. Cited by: Baselines.. Yang et al. (2024) A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Xia, X. Ren, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Wan, Y. Liu, Z. Cui, Z. Zhang, and Z. Qiu Qwen2.5 technical report. Cited by: Introduction. Yao et al. (2022a) S. Yao, H. Chen, J. Yang, and K. Narasimhan Webshop: towards scalable real-world web interaction with grounded language agents. Vol. 35. Cited by: Introduction, Benchmarks.. Yao et al. (2022b) S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao React: synergizing reasoning and acting in language models. Cited by: Baselines.. Yu et al. (2026) Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, L. Liu, et al. Dapo: an open-source llm reinforcement learning system at scale. Vol. 38. Cited by: Related Work. Zeng et al. (2026) A. Zeng, X. Lv, Z. Hou, Z. Du, Q. Zheng, B. Chen, D. Yin, C. Ge, C. Huang, C. Xie, et al. Glm-5: from vibe coding to agentic engineering. Cited by: Related Work. Zhang et al. (2025a) G. Zhang, H. Geng, X. Yu, Z. Yin, Z. Zhang, Z. Tan, H. Zhou, Z. Li, X. Xue, Y. Li, et al. The landscape of agentic reinforcement learning for llms: a survey. Cited by: Introduction. Zhang et al. (2026a) J. Zhang, K. Chen, Z. Lu, E. Zhou, Q. Yu, and J. Zhang Prune4web: dom tree pruning programming for web agent. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, p. 34710–34718. Cited by: Introduction. Zhang et al. (2026b) J. Zhang, Z. Fu, Z. Xi, W. Jing, M. Chai, W. He, G. Zhang, C. Fan, C. An, W. Chen, et al. AgentV-rl: scaling reward modeling with agentic verifier. In Findings of the Association for Computational Linguistics: ACL 2026, p. 23078–23100. Cited by: Motivation, Table 2. Zhang et al. (2026c) K. Zhang, Y. Hong, J. Bao, H. Jiang, Y. Song, H. Dingqian, and H. Xiong Gvpo: group variance policy optimization for large language model post-training. Vol. 38. Cited by: Introduction. Zhang et al. (2024) K. Zhang, J. Li, G. Li, X. Shi, and Z. Jin Codeagent: enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 13643–13658. Cited by: Introduction. Zhang et al. (2026d) W. Zhang, M. Wang, G. Liu, H. Xu, Y. Jiang, Y. Shen, G. Hou, Z. Zheng, H. Zhang, X. Li, et al. Embodied-reasoner: synergizing visual search, reasoning, and action for embodied interactive tasks. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 41178–41207. Cited by: Introduction. Zhang et al. (2025b) X. Zhang, R. Li, Z. Zhou, L. Li, Y. Qin, K. Li, X. Sun, X. Tan, C. Qu, and Y. Qi Count counts: motivating exploration in llm reasoning with count-based intrinsic rewards. Cited by: Introduction. Zhao et al. (2026) H. Zhao, S. Zhou, Y. Zhang, S. S. Yau, W. Zhang, L. Tian, T. Zhu, Y. Huang, Y. Zeng, J. Gu, et al. AEM: adaptive entropy modulation for multi-turn agentic reinforcement learning. Cited by: Introduction. Zhao et al. (2025a) X. Zhao, Z. Kang, A. Feng, S. Levine, and D. Song Learning to reason without external rewards. Cited by: Introduction. Zhao et al. (2025b) Y. Zhao, Y. Liu, J. Liu, J. Chen, X. Wu, Y. Hao, T. Lv, S. Huang, L. Cui, Q. Ye, et al. Geometric-mean policy optimization. Cited by: Related Work. Zheng et al. (2025) C. Zheng, S. Liu, M. Li, X. Chen, B. Yu, C. Gao, K. Dang, Y. Liu, R. Men, A. Yang, et al. Group sequence policy optimization. Cited by: Related Work.