Paper deep dive
SAPO: Single-Rollout Autoregressive Policy Optimization for Agentic Reinforcement Learning
Dayang Liang, Lang Feng, Bo An, Yunlong Liu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/21/2026, 3:45:28 AM
Summary
The paper introduces SAPO (Single-Rollout Autoregressive Policy Optimization), a framework for agentic reinforcement learning that integrates policy and value functions into a single autoregressive backbone. This approach eliminates the need for a separate critic model and multiple rollouts, reducing memory and computational costs while improving performance on long-horizon tasks like ALFWorld and WebShop compared to PPO and GRPO.
Entities (8)
Relation Signals (7)
SAPO → evaluatedon → WebShop
confidence 95% · Experiments across ALFWorld and WebShop...
SAPO → evaluatedon → ALFWorld
confidence 95% · Experiments across ALFWorld and WebShop...
SAPO → outperforms → GRPO
confidence 95% · SAPO trains stably and outperforms PPO and GRPO by mean +15.1 and +12.1 percentage points, respectively
SAPO → outperforms → PPO
confidence 95% · SAPO trains stably and outperforms PPO and GRPO by mean +15.1 and +12.1 percentage points, respectively
SAPO → reducesruntimeby → 33.2%
confidence 90% · reducing per-iteration runtime by 33.2% over PPO.
SAPO → usesbackbone → Qwen2.5-7B
confidence 90% · Experiments across ALFWorld and WebShop with Qwen2.5-1.5B/7B show that SAPO trains stably
SAPO → usesbackbone → Qwen2.5-1.5B
confidence 90% · Experiments across ALFWorld and WebShop with Qwen2.5-1.5B/7B show that SAPO trains stably
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Agentic reinforcement learning (RL) has become a critical stage in the post-training of large language models. Existing critic-free, group-relative methods estimate policy advantages from multiple rollouts, avoiding the substantial memory overhead of conventional proximal policy optimization (PPO) and achieving strong performance on long-horizon interactive tasks. Despite their success, recent studies revealed three limitations: (1) Lack explicit value generalization and effective temporal credit assignment; (2) Suffer from potential advantage collapse in long-horizon complex tasks; (3) Require a costly trade-off between sampling budget and policy performance. In this work, we propose Single-rollout Autoregressive Policy Optimization (SAPO), a low-memory and compute-efficient framework in which the policy and value functions share a single autoregressive backbone. SAPO exploits the autoregressive structure of LLMs to produce policy and value predictions at distinct causal boundaries with shared parameters, while independently optimizing the PPO objectives and auxiliary on-policy SARSA objectives. To robustly estimate the contribution of each turn, we further introduce a trajectory-level generalized advantage estimator that combines lambda-returns with batch normalization. Experiments across ALFWorld and WebShop with Qwen2.5-1.5B/7B show that SAPO trains stably and outperforms PPO and GRPO by mean +15.1 and +12.1 percentage points, respectively, while eliminating the memory cost of a separate critic model and reducing per-iteration runtime by 33.2% over PPO.
Tags
Links
- Source: https://arxiv.org/abs/2608.19842v1
- Canonical: https://arxiv.org/abs/2608.19842v1
Trouble viewing inline? Open PDF directly →
Full Text
44,596 characters extracted from source content.
Expand or collapse full text
SAPO: Single-Rollout Autoregressive Policy Optimization for Agentic Reinforcement Learning Dayang Liang Thanks: Equal Contribution Affiliation: Xiamen University, China Lang Feng11footnotemark: 1 Affiliation: Bo An Affiliation: Nanyang Technological University, Singaporedyliang@stu.xmu.edu.cn, lang005@e.ntu.edu.sg, ylliu@xmu.edu.cn, Yunlong Liu Thanks: Corresponding author Affiliation: Xiamen University, China Abstract Agentic reinforcement learning (RL) has become a critical stage in the post-training of large language models. Existing critic-free, group-relative methods estimate policy advantages from multiple rollouts, avoiding the substantial memory overhead of conventional proximal policy optimization (PPO) and achieving strong performance on long-horizon interactive tasks. Despite their success, recent studies revealed three limitations: (1) Lack explicit value generalization and effective temporal credit assignment; (2) Suffer from potential advantage collapse in long-horizon complex tasks; (3) Require a costly trade-off between sampling budget and policy performance. In this work, we propose Single-rollout Autoregressive Policy Optimization (SAPO), a low-memory and compute-efficient framework in which the policy and value functions share a single autoregressive backbone. SAPO exploits the autoregressive structure of LLMs to produce policy and value predictions at distinct causal boundaries with shared parameters, while independently optimizing the PPO objectives and auxiliary on-policy SARSA objectives. To robustly estimate the contribution of each turn, we further introduce a trajectory-level generalized advantage estimator that combines λ-returns with batch normalization. Experiments across ALFWorld and WebShop with Qwen2.5-1.5B/7B show that SAPO trains stably and outperforms PPO and GRPO by mean +15.1 and +12.1 percentage points, respectively, while eliminating the memory cost of a separate critic model and reducing per-iteration runtime by 33.2% over PPO. (Going On) 1 Introduction Recent advances in agentic reinforcement learning (RL) have transformed large language models from passive text generators into agents capable of long-horizon interaction such as reasoning, tool use, web search, and code execution tasks. Large-scale RL post-training has produced substantial improvements in mathematical reasoning and coding [17, 5, 7], while interactive systems such as Search-R1 demonstrate that policies can learn to acquire external information through multi-turn actions [8]. These successes suggest that learning directly from environmental feedback is becoming a critical pathway toward increasingly autonomous and general-purpose language agents. Most agentic RL approaches build on proximal policy optimization (PPO) or its critic-free, group-relative variants. Conventional PPO provides token- or turn-level credit assignment through a learned value function and generalized advantage estimation (GAE), but typically requires a policy-scale critic with substantial memory and computation budgets [13, 21]. GRPO avoids this critic by normalizing rewards across multiple sampled for the same task [17]. Nevertheless, recent studies expose three limitations. First, when all rollouts in a group receive identical or nearly identical rewards—an increasingly common event under sparse feedback and long-horizon failures—the normalized advantages vanish, causing zero-gradient or advantage-collapse regions [29]. Second, assigning a trajectory-level group advantage uniformly to all generated tokens provides weak temporal credit assignment and cannot generalize value estimates across states, prompts, or policy iterations; learned critics can identify low-value repetitive or erroneous prefixes that group-relative estimators miss [30, 31, 7]. Third, group-relative estimation entails an unfavorable compute–quality trade-off: small groups yield noisy baselines, whereas larger groups improve stability only by multiplying rollout cost and imposing synchronization barriers in long, variable-length agent trajectories [3, 6]. Several work attempts to mitigate these problems. DAPO improves group-relative learning through dynamic sampling, asymmetric clipping, and token-level policy losses, but remains dependent on multiple rollouts and group statistics [29]. ReMax and RLOO eliminate the critic using greedy or leave-one-out baselines, while VinePPO estimates intermediate values through Monte Carlo continuations; these methods reduce training-state memory but retain coarse credit assignment or introduce additional generation cost [12, 2, 9]. Conversely, Open-Reasoner-Zero, VC-PPO, and VAPO restore explicit value models and demonstrate that a well-trained critic can substantially improve long-horizon reasoning, particularly through critic pretraining and improved GAE [7, 30, 31]. However, these methods maintain a separate, often policy-scale value network. Hydra-PPO partially shares a frozen backbone, but still requires role-specific adapters and heads, and its fully shared variant reveals potential policy–value interference [15]. POISE reuses actor hidden states through a lightweight probe but requires cross-rollout estimation [3], whereas SAO enables asynchronous single-rollout training while relying on a separately pretrained critic with more frequent value updates [6]. Thus, efficient single-rollout learning with explicit temporal value estimation remains unresolved. To address this, we propose Single-Rollout Autoregressive Policy Optimization (SAPO), a lightweight RL framework that preserves value-based learning without maintaining a separate critic or relying on multiple rollouts. SAPO builds on a simple observation: the information flow required by actor–critic learning naturally follows the causal order of language generation. A value estimate should summarize the state before an action is taken, whereas evaluating that action may additionally condition on what has been generated. This alignment allows policy generation and value estimation to be expressed at appropriate positions within one autoregressive stream, with causal masking enforcing their distinct conditioning contexts. Consequently, the same backbone can generate actions and learn generalized value estimates while reusing its parameters and intermediate computation. SAPO thus reconciles two competing goals in existing approaches: it retains the explicit value generalization and temporal credit assignment of actor–critic algorithms, yet approaches the memory and computational efficiency sought by critic-free, group-relative optimization. Its central contribution is therefore not weight sharing alone, but a correspondence between autoregressive modeling and actor–critic learning that enables efficient value-based optimization within a single causal model. We evaluate SAPO on ALFWorld and WebShop using Qwen2.5-1.5B and Qwen2.5-7B backbone models. SAPO trains stably from one rollout per prompt and improves task success over conventional PPO and GRPO by +15.1 and +12.1 percentage points, respectively. It simultaneously eliminates the memory footprint of a separate critic and reduces per-iteration runtime by 33.2% relative to PPO. Our contributions are threefold. • We introduce a single-rollout autoregressive actor–critic architecture that jointly represents and trains policy and value functions through causal-boundary readouts. • We develop a trajectory-level advantage estimator that combines λ-returns with batch normalization, enabling explicit turn-level credit assignment without group-relative sampling. • We demonstrate on long-horizon agentic tasks that this unified framework achieves a favorable combination of performance, training stability, memory efficiency, and runtime efficiency. 2 Related Work Agentic RL extends reinforcement learning with verifiable rewards from single-turn response optimization to multi-turn decision making, enabling LLM agents to learn from the consequences of actions taken in external environments. A prominent line of work builds on critic-free, group-relative optimization. GRPO [17] replaces a learned value baseline with within-prompt outcome comparison, a formulation used by DeepSeek-R1 [5] and refined by DAPO [29] with dynamic sampling and asymmetric clipping, GSPO [33] with sequence-level importance ratios, and GVPO [32] through variance-aware gradient weighting. This paradigm has been extended to interactive settings: Search-R1 [8] and DeepResearcher [34] optimize iterative retrieval, ToRL [11] and WebSailor [10] train tool use and web navigation, and RAGEN [25] characterizes optimization instabilities in multi-turn training. Long-horizon tasks have also motivated more structured credit mechanisms. ARChER [35] and HiPER [14] decompose planning and execution across temporal scales, AgentGym-RL [26] progressively increases the interaction horizon, and IGPO [23] derives dense intrinsic rewards from information gain during search. Efficiency-oriented LLM RL has developed along critic-free advantage estimation, parameter sharing between policy and value functions, and systems-level execution. ReMax [12] and RLOO [2] replace learned values with greedy or leave-one-out baselines, while GRPO [17] normalizes rewards within each prompt group. Eliminating the value network reduces parameters and optimizer state, but these estimators require additional decoded trajectories and provide no value generalization across states. VinePPO [9] obtains intermediate value estimates from branched Monte Carlo continuations, likewise exchanging critic training for increased rollout computation. Parameter-sharing approaches retain explicit value learning while reducing model duplication. Hydra-PPO [15] places policy and value adapters over a frozen backbone; J-Hydra [15] further shares the trainable adapter, with potential interference between the two objectives. POISE [3] predicts returns from actor hidden states using a lightweight probe, but relies on cross-rollout construction to decouple value estimation from the evaluated trajectory. SAO [6] addresses rollout underutilization through asynchronous single-rollout collection, while retaining a separately pretrained critic with more frequent value updates. HybridFlow [18] instead improves distributed execution through model placement, resharding, and scheduling, independently of the advantage estimator. SAPO is distinguished by jointly representing the policy, state value, and action value in one causal autoregressive model, thereby eliminating both group sampling and the separate critic backbone while retaining explicit temporal credit assignment. 3 Preliminaries Multi-Turn Agentic RL. We consider RL for LLM-based agents that interact with an environment over multiple turns, where the interaction process is formulated as a finite-horizon Markov Decision Process (MDP). Given a task instance ∈p(X) x∈ p(X), at each turn t=1,2,…,Tt=1,2,…,T, an agent policy πθ _θ observes a state t∈ s_t and generates a textual action t∈ a_t , and then transitions to the next state t+1∈ s_t+1 while yielding a scalar reward rt∈ℝr_t . The interaction unfolds as a trajectory =(1,1,r1),(2,2,r2),…,(T,T,rT) τ=\( s_1, a_1,r_1),( s_2, a_2,r_2),...,( s_T, a_T,r_T)\, where T denotes the trajectory length in interaction turns. In practical tasks such as ALFWorld, the agent generates a response as its action for each observation. This response is structured within <think></think> and <action></action> tags, where the former contains the reasoning process, while the latter specifies the actual action executed in the environment. Notably, for most task settings, reward signals are sparse and delayed, e.g., the environment provides an outcome reward R()R( τ) only after the trajectory terminates. 3.1 Group-relative Policy Optimization Recent agentic RL methods for LLMs commonly adopt a group-relative policy optimization paradigm. Given a task instance x, the old policy πθold _ _ old samples a group of N candidate trajectories x=1,2,…,NG_x=\ τ_1, τ_2,…, τ_N\, where each trajectory corresponds to one complete rollout. Each trajectory i τ_i receives a scalar reward R(i)R( τ_i) that reflects the overall quality or success of the generated outcome. Instead of learning an advantage function A(t,t)A( s_t, a_t) with critic networks in PPO [16], group-based RL computes the advantage using only statistics within the sampled group: A(i)=(R(i)i=1N).A( τ_i)= GroupNormalization (\R( τ_i)\_i=1^N ). In GRPO [17], the advantage is evaluated by normalizing each trajectory reward with the mean and variance of group rewards (R(i)i=1N)(\R( τ_i)\_i=1^N). This sampling-based estimator reduces the memory and computational overhead introduced by the critic architecture in conventional PPO. 3.2 Proximal Policy Optimization Proximal Policy Optimization (PPO) is an actor-critic RL algorithm previously used for LLM post-training. Given prompts sampled from a dataset, the policy generates response trajectories and receives rewards from task-specific environments or learned reward models. PPO updates the policy by maximizing a clipped surrogate objective, ℒPPO=t[min(ρtA^t,clip(ρt,1−ϵ,1+ϵ)A^t)],L_PPO=E_t\! [ \! ( _t A_t,\,clip( _t,1-ε,1+ε) A_t ) ], where ρt=πθ(at∣st)/πθold(at∣st) _t= _θ(a_t s_t)/ _ _old(a_t s_t) is the likelihood ratio and A^t A_t is an advantage estimate. The clipping operation constrains abrupt policy changes and improves optimization stability. A learned critic Vϕ(st)V_φ(s_t) predicts expected returns, enabling advantage estimation—commonly through generalized advantage estimation—and propagating delayed rewards to earlier decisions. In LLMs, states correspond to token prefixes or interaction histories, while actions are generated tokens. Although effective for long-horizon credit assignment, PPO typically requires separate policy, critic, reference, and rollout models, resulting in substantial memory and computational overhead. 4 Method SAPO is a single-rollout actor–critic method in which one causal language model represents the policy, the state value, and the action value. As shown in Figure 1, the design follows the temporal structure of an agent turn: the model first summarizes the current interaction state, then generates a textual action, and finally evaluates that action before observing its environmental consequence. This order matches the conditioning structure required by actor–critic learning and allows all three quantities to be extracted from one autoregressive sequence. Figure 1: Framework of SAPO. We begin with introducing a two-token value basis that reads bounded scalar values from the language-model logits at two causal boundaries (Section 4.1). Second, we construct turn-level advantages and value targets by traversing each sampled trajectory backward (Section 4.2). Third, we jointly optimize the token-level policy objective and the turn-level value objectives in one actor update (Section 4.3). Algorithm 1 summarizes the complete procedure. 4.1 Autoregressive Actor–Critic Representation Causal factorization of an agent turn. At turn t, let ts c^s_t denote the tokenized context representing state t s_t, and let t=(at,1,…,at,Mt) a_t=(a_t,1,…,a_t,M_t) be the generated response. We write tsa=[ts;t] c^sa_t=[ c^s_t; a_t] for their concatenation. A valid state value may depend on ts c^s_t but not on the yet-unseen action, whereas an action value may additionally depend on the complete t a_t but not on the subsequent reward or observation. These dependencies occur naturally at the two boundaries of causal decoding: immediately before the first response token and immediately after the last response token. SAPO reads V(t)V( s_t) and Q(t,t)Q( s_t, a_t) at these respective boundaries while generating t a_t between them. The causal mask enforces the desired information separation without separate encoders or manually detached context representations. A shared two-token value basis. Let θ()∈ℝ|| z_θ( c) ^|W| be the next-token logits produced by the causal language model after context c, where W is its vocabulary. We reserve two existing vocabulary entries, w+w^+ and w−w^-, exclusively as a value basis. For any causal context, their relative evidence defines a normalized scalar readout: pθ()=clip((zθ,w+()−zθ,w−()τv),−1,1)p_θ( c)=clip (\! ( z_θ,w^+( c)-z_θ,w^-( c) _v ),-1,1 ) (1) where τv>0 _v>0 is a value temperature. For tasks whose discounted returns lie in [−Rmax,Rmax][-R_ ,R_ ], SAPO parameterizes: Vθ(t)=Rmaxpθ(ts),Qθ(t,t)=Rmaxpθ(tsa).V_θ( s_t)=R_ p_θ( c^s_t), Q_θ( s_t, a_t)=R_ p_θ( c^sa_t). (2) The logit difference makes the readout invariant to a common shift of the two logits, while the clipping supplies a learning range matched to the return scale. Importantly, w+w^+ and w−w^- are readouts rather than generated reasoning tokens: neither token is sampled, appended to the context, or revealed to the environment. Separation from the action distribution. The reserved entries must not become artificial actions. We therefore define the policy vocabulary as act=∖w+,w−W_act=W \w^+,w^-\ and normalize action probabilities only over this set: πθ(a∣)=expzθ,a()∑w∈actexpzθ,w(),a∈act. _θ(a c)= z_θ,a( c) _w _act z_θ,w( c), a _act. (3) The same restriction is applied during rollout, policy-log-probability evaluation, and entropy computation. Thus, policy learning never rewards or suppresses the value basis through the action softmax, while the raw logits of the two reserved entries remain available to the value objectives. Policy and value learning share the transformer and language-model head, but retain distinct semantics and supervision. Single-stream evaluation. For every sampled response, PPO already evaluates the actor on [ts;t][ c^s_t; a_t] to obtain old token log probabilities. SAPO extends this actor evaluation by gathering the two reserved logits at the final state-context position and the final valid action position. The former yields Vθold(t)V_ _old( s_t) and the latter yields Qθold(t,t)Q_ _old( s_t, a_t); no second model or repeated encoding of the state–action prefix is required. During optimization, the same actor forward simultaneously produces current policy log probabilities, state values, and action values. Hence, the precise computational benefit of SAPO is the removal of the independent critic forward/backward path and its training state, rather than an assumption that value estimates are obtained without any actor evaluation. 4.2 Single-Rollout Trajectory Advantage Estimation Group-relative algorithms estimate advantages by comparing several trajectories sampled for the same task. SAPO instead samples one trajectory per task and transfers delayed feedback across its interaction turns using the learned state value. All targets are computed from rewards observed in the environment and values produced by the frozen rollout policy πθold _ _old; current value predictions are never used to construct their own targets. Trajectory-wise generalized advantage estimation. For a trajectory =(t,t,rt,dt)t=1T τ=\( s_t, a_t,r_t,d_t)\_t=1^T, let dtd_t indicate termination after turn t and let Vtold=Vθold(t)V^old_t=V_ _old( s_t). We compute the temporal-difference residual and the generalized advantage recursively as: δt _t =rt+γ(1−dt)Vt+1old−Vtold, =r_t+γ(1-d_t)V^old_t+1-V^old_t, (4) AtGAE A^GAE_t =δt+γλ(1−dt)At+1GAE, = _t+γλ(1-d_t)A^GAE_t+1, (5) with AT+1GAE=0A^GAE_T+1=0. A trajectory truncated at the environment horizon is treated as a finite-horizon terminal trajectory, so its final bootstrap value is zero. This recursion gives different turns different learning signals even when the environment supplies only a terminal outcome, while λ controls the usual bias–variance trade-off. The same backward pass provides complementary targets for the two causal value boundaries. The state value is trained toward the λ-return, whereas the action value is trained toward a on-policy SARSA target: ytV=Vtold+AtGAE,ytQ=rt+γ(1−dt)Qt+1old.y^V_t=V^old_t+A^GAE_t, y^Q_t=r_t+γ(1-d_t)Q^old_t+1. (6) The first target propagates long-range outcome information to the pre-action state boundary. The second grounds the post-action boundary in the immediate consequence of the selected response. We deliberately construct the policy advantage from GAE rather than from the difference between simultaneously learned Q and V estimates: this prevents early action-value miscalibration from directly perturbing the policy while still using the Q objective to train an action-conditioned value representation. Batch-normalized turn advantages. Before assigning advantages to response tokens, we normalize them over the valid turns in the current training batch. If an environment exposes an invalid-action indicator mtinvm_t^inv, we first form A~t=AtGAE−cinvmtinv A_t=A^GAE_t-c_invm_t^inv; otherwise cinv=0c_inv=0. With batch statistics μℬ _B and σℬ _B, the policy advantage is A^t=A~t−μℬσℬ+ϵadv. A_t= A_t- _B _B+ _adv. (7) Only after this turn-level normalization do we broadcast A^t A_t to all valid tokens in t a_t. Normalizing before broadcasting prevents long responses from receiving disproportionate weight in the advantage statistics. Unlike group normalization, Equation 7 compares learning signals across independent turns and tasks; it therefore does not require multiple synchronized rollouts of the same prompt. 4.3 Joint Policy and Value Optimization Token-level policy objective. Although the environment acts at the turn level, each textual action contains multiple autoregressive decisions. Let ℓt,jold ^old_t,j and ℓt,j(θ) _t,j(θ) be the old and current log probabilities of token at,ja_t,j under the restricted action distribution in Equation 3. The importance ratio is ρt,j(θ)=exp(ℓt,j(θ)−ℓt,jold). _t,j(θ)= \! ( _t,j(θ)- ^old_t,j ). (8) Every valid response token in turn t shares A^t A_t, yielding the clipped policy loss ℒpol(θ)=−t,j[min(ρt,j(θ)A^t,clip(ρt,j(θ),1−ϵ,1+ϵ)A^t)],L_pol(θ)=-E_t,j\! [ \! ( _t,j(θ) A_t,clip( _t,j(θ),1-ε,1+ε) A_t ) ], (9) where the expectation includes only valid action tokens. This preserves PPO’s token-level trust-region surrogate while replacing both the separate critic and group-relative advantage estimator. Clipped value objectives. We optimize the value readouts in their normalized probability space. Define ptV=Vθ(t)/Rmaxp^V_t=V_θ( s_t)/R_ and ptQ=Qθ(t,t)/Rmaxp^Q_t=Q_θ( s_t, a_t)/R_ , with frozen rollout-time predictions ptV,oldp^V,old_t and ptQ,oldp^Q,old_t. Accordingly, the normalized targets are y¯tV=ytV/Rmax y^V_t=y^V_t/R_ and y¯tQ=ytQ/Rmax y^Q_t=y^Q_t/R_ with normalized range [−1,1][-1,1]. For X∈V,QX∈\V,Q\, let ptX,clip=ptX,old+clip(ptX−ptX,old,−ϵX,ϵX).p^X,clip_t=p^X,old_t+clip\! (p^X_t-p^X,old_t,- _X, _X ). (10) We use the PPO-style clipped regression loss ℒX(θ)=12t[max((ptX−y¯tX)2,(ptX,clip−y¯tX)2)],X∈V,Q.L_X(θ)= 12E_t\! [ \! ((p^X_t- y^X_t)^2,(p^X,clip_t- y^X_t)^2 ) ], X∈\V,Q\. (11) Both losses are averaged over valid turns, not over response tokens. This gives each environment decision equal weight regardless of response length. Unified objective. The final loss combines policy optimization, the two causal value objectives, reference-policy regularization, and entropy regularization: ℒSAPO(θ)=ℒpol+cVℒV+cQℒQ+βℒKL−cHℋ(πθ).L_SAPO(θ)=L_pol+c_VL_V+c_QL_Q+ _KL-c_HH( _θ). (12) The policy, state-value, and action-value terms use separate masks but update the same transformer and language-model head in one backward pass. Coefficients cVc_V and cQc_Q control interference between generation and value learning, while the KL and entropy terms regularize policy drift and exploration. This objective retains distinct supervision for the three roles without allocating role-specific model parameters. Algorithm 1 Single-Rollout Autoregressive Policy Optimization 0: Policy πθ _θ, reference policy πref _ref, task batch ℬB, γ, λ 1: for each training iteration do 2: Set θold←θ _old←θ and sample one trajectory per task with πθold _ _old 3: Evaluate sampled state–action sequences with the old actor to obtain token log probabilities, VoldV^old, and QoldQ^old 4: Traverse each trajectory backward using Equations 4–6 5: Normalize turn-level policy advantages using Equation 7 and broadcast them to valid action tokens 6: Run one joint actor forward on the sampled sequences to obtain current policy, V, and Q predictions 7: Update θ by minimizing Equation 12 8: end for In implementation, trajectory identifiers and turn indices are retained until after the backward target computation, ensuring that batching, duplication, or shuffling cannot break temporal adjacency. Reward is used only as a target after environment execution and is never included in the action-value input. Together with the causal boundary construction, this preserves the semantics of both state and action values throughout single-stream training. Type Method ALFWorld WebShop Pick Look Clean Heat Cool Pick2 All Score Succ. Closed-Source Model Prompting GPT-4o 75.3 60.8 31.2 56.7 21.6 49.8 48.0 31.8 23.7 Prompting Gemini-2.5-Pro 92.8 63.3 62.1 69.0 26.6 58.7 60.3 42.5 35.9 Qwen2.5-1.5B-Instruct Prompting ReAct 17.4 20.5 15.7 6.2 7.7 2.0 12.8 40.1 11.3 Prompting Reflexion 35.3 22.2 21.7 13.6 19.4 3.7 21.8 55.8 21.9 RL Training RLOO 88.3±3.0 52.8±8.6 71.0±5.9 62.8±8.7 66.4±5.5 56.9±4.7 69.7±2.5 73.9±5.6 52.1±6.7 RL Training EMPG 85.5 33.5 78.9 76.2 74.7 89.1 73.7 80.4 60.8 RL Training GiGPOw/ std 94.4±5.9 67.5±4.6 94.8±3.8 94.4±7.8 79.8±4.7 76.4±5.4 86.7±1.7 83.1±1.6 65.0±3.2 RL Training GiGPOw/o std 96.0±1.4 76.5±3.9 91.8±5.5 91.3±6.3 71.7±8.4 79.5±7.7 86.1±4.7 83.5±1.8 67.4±4.5 RL Training PPO (with critic) 64.8±3.5 40.5±6.9 57.1±4.9 60.6±6.6 46.4±4.0 47.4±1.9 54.4±3.1 73.8±3.0 51.5±2.9 RL Training GRPO 85.3±1.5 53.7±8.0 84.5±6.8 78.2±7.9 59.7±5.0 53.5±5.6 72.8±3.6 75.8±3.5 56.8±3.8 RL Training SAPO 92.0±2.9 76.9±6.3 100.0±0.0 100.0±0.0 82.8±4.7 82.4±5.0 90.1±2.3 82.21.4 63.71.6 Qwen2.5-7B-Instruct Prompting ReAct 48.5 35.4 34.3 13.2 18.2 17.6 31.2 46.2 19.5 Prompting Reflexion 62.0 41.6 44.9 30.9 36.3 23.8 42.7 58.1 28.8 RL Training RLOO 87.6±4.3 78.2±8.3 87.3±5.8 81.3±7.6 71.9±5.2 48.9±8.4 75.5±4.6 80.3±3.2 65.7±4.0 RL Training EMPG 92.9 75.2 74.8 86.3 73.7 65.3 78.5 81.0 69.3 RL Training GiGPOw/ std 97.7±1.6 82.7±7.9 98.8±1.6 83.7±7.2 89.3±8.2 79.2±6.6 90.8±1.3 84.4±2.9 72.8±3.2 RL Training GiGPOw/o std 91.8±5.4 88.6±6.3 95.9±3.2 90.2±2.6 86.5±5.5 85.2±7.5 90.2±2.3 86.2±2.6 75.2±3.8 RL Training PPO (with critic) 92.3±4.0 64.0±8.4 92.5±2.4 89.5±7.0 80.3±2.0 68.8±8.3 80.4±2.7 81.4±3.1 68.7±5.1 RL Training GRPO 90.8±5.1 66.1±6.7 89.3±5.4 74.7±6.9 72.5±5.4 64.7±7.3 77.6±5.2 79.3±2.8 66.1±3.7 RL Training SAPO 99.0±1.4 82.3±2.1 100.0±0.0 97.9±4.7 79.7±3.9 91.7±1.6 94.0±1.7 88.6±1.8 82.4±2.0 Table 1: Evaluation results on ALFWorld and WebShop. For each RL training method, we report the mean and standard deviation over three random seeds. The ALFWorld contains six categories: Pick & Place (Pick), Examine in Light (Look), Clean & Place (Clean), Heat & Place (Heat), Cool & Place (Cool), and Pick Two & Place (Pick2). Most entries in this table are reported by Feng et al. [4]. Notably, the baseline GiGPOw/o std replaces the group-relative normalization std with one. 5 Experiments We evaluate SAPO across a range of multi-turn environments. Specifically, our experiments aim to answer four research questions: (1) How does SAPO perform compared with the core PPO and GRPO baselines? (2) How sensitive is SAPO to its components, and how much does each component contribute? (3) How does SAPO compare with a comparable PPO baseline in terms of resource overhead? (4) Can SAPO effectively maintain stable performance over continuous training iterations over long-horizon tasks? 5.1 Experimental Setup Benchmarks and Baselines. We train and evaluate SAPO on two challenging multi-turn ALFWorld [20] and WebShop [27] benchmarks. ALFWorld is an embodied household environment for evaluating long-horizon textual reasoning and decision-making. In each episode, the agent receives a concrete task instruction sampled from 3,8273,827 tasks from six categories. WebShop is an interactive web-shopping environment containing nearly 1.11.1 million products and 12,00012,000 user instructions. We compare SAPO against three categories of competitive baselines: (1) Proprietary models: GPT-4o [1] and Gemini-2.5-Pro [22]; (2) training-free prompting agents: ReAct [28] and Reflexion [19]; and (3) RL-based methods: PPO [16], RLOO [2], GRPO [17], EMPG [24] and GiGPO[4]. Implementation details. We employ Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct as the base models for training experiments. To ensure fair comparisons, the hyperparameter settings of SAPO follow the existing RL framework [4] in each benchmark. Specifically, we use a learning rate of 1×10−61× 10^-6, and a KL-penalty coefficient of 0.010.01. The maximum number of interaction turns is set to 5050 for ALFWorld and 1515 for WebShop, and we train for 150150 steps overall. All our experiments were run on 4 NVIDIA H200s and 8 NVIDIA A40s. The technical supplement provides additional implementation details. 5.2 Experiment Results Figure 2: Per-iteration runtime breakdown of PPO and SAPO on ALFWorld using Qwen2.5-1.5B. The vertical axis is shown on a logarithmic scale to accommodate the large variation in module runtimes. N/A indicates that SAPO does not require the corresponding value-model or critic-update module. Overall, SAPO reduces the measured per-iteration runtime from 451.2 s to 301.4 s, corresponding to a 33.2% reduction compared with PPO. As shown in Table 1, across ALFWorld and WebShop with Qwen2.5-1.5B/7B, SAPO consistently outperforms standard PPO and GRPO at both model scales, while matching or exceeding recent improved variants overall, with particularly clear gains at Qwen2.5-7B. For example, with Qwen2.5-1.5B, SAPO achieves 90.1%90.1\% aggregate success on ALFWorld, improving over PPO and GRPO by 35.735.7 and 17.317.3 percentage points, respectively, and reaches perfect performance on the Clean and Heat categories. On WebShop, it improves PPO by 8.48.4 points in score and 12.212.2 points in success rate. The highlighted advantage persists at 7B, i.e., SAPO obtains 94.0%94.0\% ALFWorld success and an 88.688.6 WebShop score with 82.4%82.4\% success, outperforming both PPO and GRPO as well as the recent variants on all three aggregate metrics. Although SAPO is not uniformly best on every individual category, its aggregate performance is consistently best. Overall, the results indicate that SAPO’s gains are not confined to a specific model capacity or environment. By integrating value estimation and policy optimization within a single-rollout autoregressive process, SAPO retains effective temporal credit assignment without a separate critic or group-based sampling. Its strong performance using sparse outcome rewards further suggests that the architectural alignment between autoregressive modeling and actor–critic learning can reduce dependence on task-specific dense reward engineering. 5.3 Per-iteration Runtime Comparison Figure 2 compares the per-iteration runtime breakdown of PPO and SAPO on ALFWorld using Qwen2.5-1.5B. SAPO reduces the total measured runtime from 451.2 s to 301.4 s, yielding a 33.2% reduction over PPO. The largest absolute saving comes from trajectory generation, whose runtime decreases from 306.4 s to 221.4 s. More importantly, SAPO eliminates the separate value inference and critic optimization stages, which together account for 61.4 s per iteration in PPO. In contrast, the remaining actor-side costs are comparable: old-policy log-probability computation takes 14.3 s versus 14.2 s, reference-model evaluation takes 13.3 s versus 12.3 s, and actor updates take 54.2 s versus 52.6 s for PPO and SAPO, respectively. These results indicate that integrating value estimation into the shared autoregressive model introduces little additional actor-side overhead. Overall, SAPO improves runtime efficiency structurally by removing the independent critic pathway while retaining explicit value learning and temporal credit assignment. 6 Conclusion We introduced SAPO, a single-rollout autoregressive actor–critic framework for RL of LLM agents. SAPO exploits causal boundaries within one language model to represent the policy, state value, and action value without a separate critic network or group-relative sampling. The unified objective jointly optimizes policy and value learning through a shared backbone while preserving their distinct conditioning and supervision. Experiments on ALFWorld and WebShop with Qwen2.5-1.5B and 7B demonstrate stable training and task performance, outperforming PPO and GRPO while using only one rollout per task. SAPO also removes the memory cost of a policy-scale critic and reduces measured per-iteration runtime by 33.2% relative to PPO. These results show that explicit value learning need not require duplicated models or costly sampling, offering a path toward efficient long-horizon agentic RL. References [1] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. (2023) Gpt-4 technical report. Cited by: §5.1. [2] A. Ahmadian, C. Cremer, M. Gallé, M. Fadaee, J. Kreutzer, O. Pietquin, A. Üstün, and S. Hooker (2024) Back to basics: revisiting REINFORCE-style optimization for learning from human feedback in LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand, p. 12248–12267. External Links: Document, Link Cited by: §1, §2, §5.1. [3] Y. Choi, J. Lim, W. Ahn, M. Oh, J. Shim, and Y. Jo (2026) Your language model is its own critic: reinforcement learning with value estimation from actor’s internal states. External Links: 2605.07579, Document, Link Cited by: §1, §1, §2. [4] L. Feng, Z. Xue, T. Liu, and B. An (2026) Group-in-group policy optimization for llm agent training. Vol. 38. Cited by: Table 1, §5.1, §5.1. [5] D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, X. Zhang, X. Yu, Y. Wu, Z. F. Wu, Z. Gou, Z. Shao, Z. Li, Z. Gao, et al. (2025) DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature 645, p. 633–638. External Links: Document, Link Cited by: §1, §2. [6] Z. Hou, Y. Li, J. Tang, and Y. Dong (2026) Single-rollout asynchronous optimization for agentic reinforcement learning. External Links: 2607.07508, Document, Link Cited by: §1, §1, §2. [7] J. Hu, Y. Zhang, Q. Han, D. Jiang, X. Zhang, and H. Shum (2025) Open-Reasoner-Zero: an open source approach to scaling up reinforcement learning on the base model. In Advances in Neural Information Processing Systems, Vol. 38, p. 162239–162262. External Links: Document, Link Cited by: §1, §1, §1. [8] B. Jin, H. Zeng, Z. Yue, J. Yoon, S. Arik, D. Wang, H. Zamani, and J. Han (2025) Search-R1: training LLMs to reason and leverage search engines with reinforcement learning. External Links: 2503.09516, Document, Link Cited by: §1, §2. [9] A. Kazemnejad, M. Aghajohari, E. Portelance, A. Sordoni, S. Reddy, A. Courville, and N. Le Roux (2024) VinePPO: unlocking RL potential for LLM reasoning through refined credit assignment. External Links: 2410.01679, Document, Link Cited by: §1, §2. [10] K. Li, Z. Zhang, H. Yin, L. Zhang, L. Ou, J. Wu, W. Yin, B. Li, Z. Tao, X. Wang, et al. (2025) WebSailor: navigating super-human reasoning for web agent. Cited by: §2. [11] X. Li, H. Zou, and P. Liu (2025) ToRL: scaling tool-integrated reinforcement learning. Cited by: §2. [12] Z. Li, T. Xu, Y. Zhang, Z. Lin, Y. Yu, R. Sun, and Z. Luo (2024) ReMax: a simple, effective, and efficient reinforcement learning method for aligning large language models. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235, p. 29128–29163. External Links: Link Cited by: §1, §2. [13] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe (2022) Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, Vol. 35, p. 27730–27744. External Links: Document, Link Cited by: §1. [14] J. Peng, Y. Liu, R. Zhou, C. Fleming, Z. Wang, A. Garcia, and M. Hong (2026) HiPER: hierarchical plan–execute reinforcement learning for multi-turn LLM agents. In Forty-third International Conference on Machine Learning, Cited by: §2. [15] M. Santacroce, Y. Lu, H. Yu, Y. Li, and Y. Shen (2023) Efficient RLHF: reducing the memory usage of PPO. External Links: 2309.00754, Document, Link Cited by: §1, §2. [16] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. External Links: 1707.06347, Document, Link Cited by: §3.1, §5.1. [17] Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo (2024) DeepSeekMath: pushing the limits of mathematical reasoning in open language models. External Links: 2402.03300, Document, Link Cited by: §1, §1, §2, §2, §3.1, §5.1. [18] G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu (2025) HybridFlow: a flexible and efficient RLHF framework. In Proceedings of the Twentieth European Conference on Computer Systems, p. 1279–1297. Cited by: §2. [19] N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao (2023) Reflexion: language agents with verbal reinforcement learning. Vol. 36. Cited by: §5.1. [20] M. Shridhar, X. Yuan, M. Côté, Y. Bisk, A. Trischler, and M. Hausknecht (2020) Alfworld: aligning text and embodied environments for interactive learning. Cited by: §5.1. [21] N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano (2020) Learning to summarize with human feedback. In Advances in Neural Information Processing Systems, Vol. 33, p. 3008–3021. External Links: Link Cited by: §1. [22] G. Team, R. Anil, S. Borgeaud, J. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al. (2023) Gemini: a family of highly capable multimodal models. Cited by: §5.1. [23] G. Wang, S. Dai, G. Ye, Z. Gan, W. Yao, Y. Deng, X. Wu, and Z. Ying (2026) Information gain-based policy optimization: a simple and effective approach for multi-turn search agents. In The Fourteenth International Conference on Learning Representations, Cited by: §2. [24] J. Wang, J. Liu, Y. Fu, Y. Li, X. Wang, Y. Lin, Y. Yue, L. Zhang, Y. Wang, and W. KE (2026) Harnessing uncertainty: entropy-modulated policy gradients for long-horizon LLM agents. In Forty-third International Conference on Machine Learning, Cited by: §5.1. [25] Z. Wang, K. Wang, Q. Wang, P. Zhang, L. Li, Z. Yang, X. Jin, K. Yu, M. N. Nguyen, L. Liu, et al. (2025) RAGEN: understanding self-evolution in LLM agents via multi-turn reinforcement learning. Cited by: §2. [26] Z. Xi, J. Huang, C. Liao, B. Huang, J. Liu, H. Guo, Y. Yang, R. Zheng, J. Ye, J. Zhang, et al. (2026) AgentGym-RL: an open-source framework to train LLM agents for long-horizon decision making via multi-turn reinforcement learning. In The Fourteenth International Conference on Learning Representations, Cited by: §2. [27] S. Yao, H. Chen, J. Yang, and K. Narasimhan (2022) Webshop: towards scalable real-world web interaction with grounded language agents. Vol. 35. Cited by: §5.1. [28] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao (2022) React: synergizing reasoning and acting in language models. Cited by: §5.1. [29] Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, L. Liu, X. Liu, H. Lin, Z. Lin, B. Ma, G. Sheng, Y. Tong, C. Zhang, M. Zhang, W. Zhang, H. Zhu, J. Zhu, J. Chen, J. Chen, C. Wang, H. Yu, Y. Song, X. Wei, H. Zhou, J. Liu, W. Ma, Y. Zhang, L. Yan, M. Qiao, Y. Wu, and M. Wang (2025) DAPO: an open-source LLM reinforcement learning system at scale. External Links: 2503.14476, Document, Link Cited by: §1, §1, §2. [30] Y. Yuan, Y. Yue, R. Zhu, T. Fan, and L. Yan (2025) What’s behind PPO’s collapse in long-CoT? value optimization holds the secret. External Links: 2503.01491, Document, Link Cited by: §1, §1. [31] Y. Yue, Y. Yuan, Q. Yu, X. Zuo, R. Zhu, W. Xu, J. Chen, C. Wang, T. Fan, Z. Du, X. Wei, X. Yu, G. Liu, J. Liu, L. Liu, H. Lin, Z. Lin, B. Ma, C. Zhang, M. Zhang, W. Zhang, H. Zhu, R. Zhang, X. Liu, M. Wang, Y. Wu, and L. Yan (2025) VAPO: efficient and reliable reinforcement learning for advanced reasoning tasks. External Links: 2504.05118, Document, Link Cited by: §1, §1. [32] K. Zhang, Y. Hong, J. Bao, H. Jiang, Y. Song, H. Dingqian, and H. Xiong (2025) GVPO: group variance policy optimization for large language model post-training. External Links: 2504.19599, Document, Link Cited by: §2. [33] C. Zheng, S. Liu, M. Li, X. Chen, B. Yu, C. Gao, K. Dang, Y. Liu, R. Men, A. Yang, et al. (2025) Group sequence policy optimization. Cited by: §2. [34] Y. Zheng, D. Fu, X. Hu, X. Cai, L. Ye, P. Lu, and P. Liu (2025) DeepResearcher: scaling deep research via reinforcement learning in real-world environments. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 414–431. Cited by: §2. [35] Y. Zhou, A. Zanette, J. Pan, S. Levine, and A. Kumar (2024) ARChER: training language model agents via hierarchical multi-turn reinforcement learning. Cited by: §2.