Paper deep dive
OGPO: Offline Goal-conditioned Policy Optimization for Recoverable Vision-Language-Action Models
Xule Gao, Xi Wang, Zehua Zang, Rui Wang, Changwen Zheng, Chuxiong Sun
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 10/12/2026, 6:31:26 AM
Summary
The paper introduces OGPO, an Offline Goal-conditioned Policy Optimization framework designed to enhance the recoverability of Vision-Language-Action (VLA) models in robotic manipulation. Unlike standard Supervised Fine-Tuning (SFT) which imitates step-wise actions, OGPO adaptively relabels offline trajectories with semantic key states and constructs goal-conditioned process rewards. This shifts training from trajectory imitation to key-state reachability learning, allowing policies to recover from execution deviations without online interaction. Experiments on LIBERO, MetaWorld, and the proposed LIBERO-ReAct benchmark demonstrate that OGPO improves robustness and goal-directed recoverability under distribution shifts and perturbations.
Entities (12)
Relation Signals (8)
OGPO → improves → VLA policy performance
confidence 95% · OGPO consistently improves VLA policy performance over standard supervised fine-tuning.
OGPO → uses → Goal-Conditioned Process Rewards
confidence 94% · OGPO adaptively relabels trajectories with semantic key states and constructs goal-conditioned process rewards
OGPO → shiftstrainingfrom → trajectory-level action imitation
confidence 93% · OGPO shifts VLA post-training from trajectory-level action imitation to key-state reachability learning
OGPO → shiftstrainingto → key-state reachability learning
confidence 93% · OGPO shifts VLA post-training from trajectory-level action imitation to key-state reachability learning
LIBERO-ReAct → evaluates → goal-directed recoverability
confidence 92% · LIBERO-ReAct... is designed to evaluate goal-directed recoverability during policy execution.
$\\pi_{0.5}$ → isbackbonefor → OGPO
confidence 90% · OGPO is built on the pi0 and pi0.5 backbones.
pi0 → isbackbonefor → OGPO
confidence 90% · OGPO is built on the pi0 and pi0.5 backbones.
Supervised Fine-tuning → isbaselinefor → OGPO
confidence 90% · OGPO consistently improves VLA policy performance over standard supervised fine-tuning.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Vision-language-action (VLA) models have shown strong promise in robotic manipulation, yet their post-training still predominantly relies on trajectory-level supervised imitation over robot demonstrations. While effective for learning feasible actions, this paradigm supervises policies to reproduce expert actions at each step, biasing them toward demonstration-specific action paths and providing limited guidance for recovering toward task-progress states once execution deviates from demonstrated trajectories. To address this limitation, we propose OGPO, an Offline Goal-conditioned Policy Optimization framework for recoverable VLA post-training. Instead of treating offline demonstrations merely as step-wise action labels, OGPO adaptively relabels trajectories with semantic key states and constructs goal-conditioned process rewards to optimize the policy for reaching these task-progress states. In this way, OGPO shifts VLA post-training from trajectory-level action imitation to key-state reachability learning, enabling policies to acquire a more robust ability to complete goal states from feasible off-demonstration states without requiring additional online interaction. Experiments on LIBERO and MetaWorld show that OGPO consistently improves VLA policy performance over standard supervised fine-tuning. Moreover, evaluations on LIBERO-Plus and our proposed LIBERO-ReAct, a mid-execution perturbation benchmark, further demonstrate that OGPO achieves stronger robustness and goal-directed recoverability under distribution shifts and recoverable off-demonstration states.
Tags
Links
- Source: https://arxiv.org/abs/2610.12914v1
- Canonical: https://arxiv.org/abs/2610.12914v1
Trouble viewing inline? Open PDF directly →
Full Text
62,280 characters extracted from source content.
Expand or collapse full text
OGPO: Offline Goal-conditioned Policy Optimization for Recoverable Vision-Language-Action Models Xule Gao Affiliation: National Key Laboratory of Space Integrated Information System Affiliation: Institute of Software, Chinese Academy of Sciences Affiliation: University of Chinese Academy of Sciences Xi Wang Affiliation: National Key Laboratory of Space Integrated Information System Affiliation: Institute of Software, Chinese Academy of Sciences Affiliation: University of Chinese Academy of Sciences Zehua Zang Affiliation: Wangxuan Institute of Computer Technology, Peking University Rui Wang Affiliation: National Key Laboratory of Space Integrated Information System Affiliation: Institute of Software, Chinese Academy of Sciences Changwen Zheng Affiliation: National Key Laboratory of Space Integrated Information System Affiliation: Institute of Software, Chinese Academy of Sciences Chuxiong Sun Email: gaoxule24@mails.ucas.ac.cn† Corresponding author: chuxiong2016@iscas.ac.cn Affiliation: National Key Laboratory of Space Integrated Information System Affiliation: Institute of Software, Chinese Academy of Sciences Abstract Vision-language-action (VLA) models have shown strong promise in robotic manipulation, yet their post-training still predominantly relies on trajectory-level supervised imitation over robot demonstrations. While effective for learning feasible actions, this paradigm supervises policies to reproduce expert actions at each step, biasing them toward demonstration-specific action paths and providing limited guidance for recovering toward task-progress states once execution deviates from demonstrated trajectories. To address this limitation, we propose OGPO, an Offline Goal-conditioned Policy Optimization framework for recoverable VLA post-training. Instead of treating offline demonstrations merely as step-wise action labels, OGPO adaptively relabels trajectories with semantic key states and constructs goal-conditioned process rewards to optimize the policy for reaching these task-progress states. In this way, OGPO shifts VLA post-training from trajectory-level action imitation to key-state reachability learning, enabling policies to acquire a more robust ability to complete goal states from feasible off-demonstration states without requiring additional online interaction. Experiments on LIBERO and MetaWorld show that OGPO consistently improves VLA policy performance over standard supervised fine-tuning. Moreover, evaluations on LIBERO-Plus and our proposed LIBERO-ReAct, a mid-execution perturbation benchmark, further demonstrate that OGPO achieves stronger robustness and goal-directed recoverability under distribution shifts and recoverable off-demonstration states. 1 Introduction Vision-language-action (VLA) policies have shown strong potential for language-conditioned robotic manipulation by scaling supervised action prediction on large-scale robot datasets [3, 40, 17]. Most existing VLA policies are adapted to downstream tasks through supervised fine-tuning (SFT), where the model is optimized to predict expert actions from visual observations and language instructions on demonstration trajectories [16, 30, 2, 1]. Although this paradigm is simple and stable, it treats demonstrations primarily as step-wise action labels for trajectory-level imitation. As a result, policies trained purely with SFT tend to reproduce demonstration-specific action paths and remain inherently bounded by the coverage and quality of demonstration data. As shown in Figure 1, when execution deviates from demonstrated trajectories, such policies may suffer from compounding errors in long-horizon tasks and lack explicit guidance for recovering toward task-progress states [28]. Recent studies have explored reinforcement learning (RL) to further improve pretrained VLA policies beyond supervised imitation, including iterative online RL fine-tuning [11], PPO/GRPO-style policy-gradient optimization [24, 19, 5], residual policy learning [34], and RL-based policy distillation [15]. By optimizing task-level rewards, RL can in principle encourage goal-directed behaviors that are not directly captured by step-wise action imitation. However, existing RL-based VLA post-training methods typically require costly environment rollouts and suffer from limited sample efficiency, making them difficult to scale for real-world robotic post-training. In this work, we study whether VLA policies can acquire goal-directed recoverability purely from offline robot data, without requiring additional online interaction. We define this ability as recovering toward task-progress key states when execution deviates from demonstrated trajectories. Our key insight is that offline demonstrations should not be used merely as action sequences to imitate; they also contain implicit evidence about which task-progress states are reachable and which behaviors move the policy toward them. Based on this insight, we propose OGPO, an offline goal-conditioned policy optimization framework for recoverable VLA post-training. Specifically, we retain SFT as a lightweight action prior to preserve low-level action feasibility and language-action alignment, while introducing an offline goal-conditioned RL objective to learn key-state reachability from existing trajectories. To turn offline data into goal-reaching supervision, OGPO adaptively relabels trajectories with task-progress key states and constructs goal-conditioned process rewards to guide both policy optimization and value estimation. Instead of supervising the policy to imitate every action along the demonstration path, this objective trains the policy to reach task-progress states under goal-conditioned rewards. In this way, OGPO shifts VLA post-training from trajectory-level action imitation to key-state reachability learning, enabling policies to improve goal-directed recoverability under feasible off-demonstration deviations while avoiding the high interaction cost of online RL. (a) Put the wine bottle on top of the cabinet. (b) Open the top of drawer and put the bowl inside. Figure 1: We design two representative perturbation cases to evaluate the robustness and goal-directed recoverability of offline-trained VLA policies under deviations from demonstration trajectories. In case (a), the gripper is forcibly opened after grasping the object, while position perturbation is applied when the end effector approaches the drawer. The results show that π0.5 _0.5-SFT fails to complete the task, whereas π0.5 _0.5-OGPO successfully recovers toward the goal state and finishes the task, demonstrating stronger recoverability under execution-time deviations. We evaluate OGPO on LIBERO [21] and MetaWorld [35], and further test robustness and recoverability on LIBERO-Plus [9] and our proposed LIBERO-ReAct benchmark with mid-execution perturbations. The results show that OGPO consistently improves over SFT baselines, especially on long-horizon and difficult tasks, while achieving stronger robustness and goal-directed recoverability under distribution shifts and execution-time deviations. Our contributions are threefold. • First, we formulate VLA post-training from the perspective of goal-directed recoverability and identify the limitation of trajectory-level SFT in recovering toward task-progress states under execution deviations. • Second, we propose OGPO, an offline goal-conditioned policy optimization framework that adaptively relabels fixed robot trajectories with task-progress key states and constructs goal-conditioned process rewards, allowing the policy to learn and optimize goal-directed behaviors beyond step-wise action imitation. • Third, we conduct extensive experiments on standard manipulation, distribution-shift, and mid-execution perturbation benchmarks, demonstrating that OGPO consistently improves task performance, robustness, and goal-directed recoverability without requiring additional online interaction. 2 Related Works VLA post-training aims to adapt pretrained vision-language-action policies to downstream robotic manipulation tasks while preserving their general action priors and improving task-specific performance. Existing approaches mainly follow two directions: SFT and RL. 2.1 Supervised Fine-tuning for VLA Policies SFT has become a dominant post-training strategy for adapting pretrained VLA policies to downstream robotic tasks. In this paradigm, the model is fine-tuned on expert demonstration trajectories by minimizing a supervised action prediction loss conditioned on visual observations and language instructions. Existing VLA models instantiate this paradigm with different action representations and prediction objectives. OpenVLA [17] discretizes continuous robot actions into action tokens and optimizes the policy with a next-token prediction objective using cross-entropy loss. Octo [30] predicts continuous action chunks through a diffusion-based action head trained with a diffusion loss. π0 _0 [2] and π0.5 _0.5 [1] further adopt flow-matching objectives for continuous action-chunk prediction. Recent studies such as OpenVLA-OFT [16] improve SFT-based adaptation by introducing continuous action representations, action chunking, parallel decoding, and regression-based imitation objectives. Despite these advances, SFT remains inherently constrained by the coverage and quality of demonstration data: it primarily learns to reproduce expert actions along demonstrated trajectories, and thus provides limited support for generalizing to unseen states, handling execution perturbations, and mitigating long-horizon compounding errors [28]. 2.2 Reinforcement Learning for VLA Post-training Recent works have explored RL as an additional post-training stage after SFT. By optimizing task-level rewards through online rollouts, RL allows VLA policies to move beyond the behavior distribution of expert demonstrations and potentially acquire more robust behaviors under distribution shift [22]. For example, iRe-VLA [11] alternates between online RL and supervised learning to improve policy performance while maintaining training stability. Other methods apply PPO- or GRPO-style policy-gradient optimization to VLA models [24, 19, 5], aiming to enhance long-horizon decision-making and task success through outcome-driven rewards. In addition, residual RL methods improve pretrained VLA policies by learning lightweight corrective policies around frozen generalist models [34], while RL-based policy distillation methods leverage VLA teachers or RL experts to obtain more specialized policies [15]. More recent studies further explore simulator- or world-model-based reinforcement fine-tuning to reduce the cost of direct real-world interaction [33, 20, 38, 14]. Although these methods demonstrate the promise of RL for improving VLA policies beyond SFT, they typically rely on online rollouts, real-world interaction, or simulator-based policy evaluation, making them costly and sample-inefficient for long-horizon robotic tasks. Our distinction. In this work, we combine the benefits of SFT and RL in a fully offline VLA post-training framework. Different from prior methods that rely mainly on action imitation, online rollouts, or simulator-based policy evaluation, OGPO extracts goal-reaching and failure-aware supervision directly from offline robot data. This enables VLA policies to improve goal-directed recoverability without additional environment interaction. To the best of our knowledge, OGPO is the first framework that integrates SFT regularization with offline goal-conditioned reinforcement learning for recoverable VLA post-training. 3 Method In this section, we present OGPO, an offline goal-conditioned policy optimization framework for recoverable VLA models. OGPO operates in three stages. First, it relabels offline trajectories into key-state-centered goal-reaching segments, thereby exposing the task-progress structure hidden in demonstrations. Second, it converts segment-level completion labels into goal-conditioned process rewards, which provide dense learning signals for value estimation. Third, it performs offline advantage-weighted policy optimization with SFT regularization, enabling the policy to improve toward goal-reaching behaviors while preserving feasible action priors. Figure 2: Offline Goal-conditioned Policy Optimization(OGPO) framework. 3.1 Goal-Relabeled Trajectory Segmentation To discover the task-progress structure embedded in offline trajectories, OGPO first segments each trajectory according to gripper state change. Given an expert demonstration trajectory τ: τ=(ot,at)t=1T,lτ= \\(o_t,a_t)\_t=1^T,l \ (1) where oto_t and ata_t denote the observation and action at timestep t, and l is a natural language instruction describing the task carried out in the demonstration. We assume that each trajectory τ is composed of a sequence of task-progress stages. The objective of trajectory segmentation is to split the trajectory into non-overlapping temporal segments τ1,τ2,…,τM\τ^1,τ^2,…,τ^M\, each corresponding to a task-progress stage. Following Chen et al. [39] and Zhu et al. [7], we realize such segmentation based on keyframes containing the change of gripper state. Despite its simplicity, gripper state change is a robust indicator of critical physical interaction, which can be viewed as a signal of the end of one stage and the beginning of the next. After segmentation, we perform goal relabeling for each segment. Specifically, we first assign each segment τmτ^m a semantic local goal gmg^m based on the task instruction, gripper transitions, and observed state changes, as detailed in Appendix B.2. We then assign each segment a binary completion label ym=[τm achieves gm]y^m=I[τ^m achieves g^m], using different labeling procedures for successful and failed trajectories: • For a successful trajectory, each valid segment is treated as completing its assigned local goal and is therefore labeled with ym=1y^m=1. The segment-end observation is regarded as the corresponding achieved task-progress key state. • For a failed trajectory, the task-level outcome does not directly indicate which local goals have been completed, as meaningful intermediate progress may occur before the eventual failure. We therefore introduce an offline VLM-based annotator to evaluate each segment and assign a binary completion label ymy^m. Details of the completion labeling are provided in Appendix B.3 and B.4. After relabeling, we obtain a goal-relabeled offline dataset: ~=(ot,l,at,gt,yt). D=\(o_t,l,a_t,g_t,y_t)\. (2) Through this process, we obtain a diverse goal-relabeled dataset that contains both achieved and unfulfilled task-progress key states, providing a solid foundation for subsequent VLA policy optimization. 3.2 Designing Goal-Conditioned Process Rewards for Value Learning Directly constructing a goal-conditioned policy on top of a pretrained VLA model would require architectural modifications and may interfere with the pretrained vision-language representations. To avoid this, we use goal-conditioned process rewards as training-time signals for offline policy optimization. This design preserves the original VLA policy structure and instruction-conditioned action interface, while still enabling the policy to acquire goal-directed behaviors by optimizing toward task-progress key states. Concretely, we construct goal-conditioned process rewards from the relabeled key states and completion labels. The relabeled goal gtg_t is used only to annotate rewards and define segment-level goal progress during training; it is not introduced as an additional input to either the policy or the value function. For each relabeled segment τm=(ot,l,at,gt,yt)t=smemτ^m=\(o_t,l,a_t,g_t,y_t)\_t=s_m^e_m, where sms_m and eme_m denote the start and end time steps of the segment, we define a normalized progress variable: ρtm=yt⋅t−smem+1−sm,t∈[sm,em+1]. _t^m=y_t· t-s_me_m+1-s_m, t∈[s_m,e_m+1]. (3) When yt=1y_t=1, ρtm _t^m increases from 00 to 11 along the segment, providing dense progress reward for a completed goal. When yt=0y_t=0, ρtm=0 _t^m=0 for the entire segment, so failed segments do not receive progressive positive rewards. For each transition t∈[sm,em]t∈[s_m,e_m], we use a segment-end indicator δtm=[t=em] _t^m=I[t=e_m], which equals 1 only at the final transition of the segment and 0 otherwise. The reward is then annotated as: rt=−λstep+Rprog(ρt+1m−ρtm)+δtm(Rsuccessgoalyt−Rfailgoal(1−yt)).r_t=- _step+R^prog ( _t+1^m- _t^m )+ _t^m (R^goal_successy_t-R^goal_fail(1-y_t) ). (4) The first term is a small step penalty that discourages redundant actions. The second term assigns dense positive credit to transitions inside achieved key-state segments. The last term provides a terminal bonus when the assigned key state is achieved and a terminal penalty when the key state remains unfulfilled. The reward-augmented offline dataset is written as: ~=(ot,l,at,gt,yt,rt). D=\(o_t,l,a_t,g_t,y_t,r_t)\. (5) With the annotated rewards, we train a value function Vψ(ot,l)V_ψ(o_t,l) to estimate the expected discounted return induced by the key-state relabeled rewards. For value learning, we use an n-step Bellman backup to construct the temporal-difference target: V^t=∑i=0n−1γirt+i+γnVψ¯(ot+n,l), V_t= _i=0^n-1γ^ir_t+i+γ^nV_ ψ(o_t+n,l), (6) where Vψ¯V_ ψ denotes a target value network or a stop-gradient copy of the value model. In implementation, VψV_ψ is obtained by attaching a lightweight value head to the intermediate representation of the VLA backbone. The value function is optimized by minimizing the temporal-difference error: ℒV(ψ)=(ot,l)∼~[(Vψ(ot,l)−V^t)2].L_V(ψ)=E_(o_t,l) D [ (V_ψ(o_t,l)- V_t )^2 ]. (7) Through this objective, states that are followed by achieved key-state progress are assigned higher values, while states in failed or incomplete segments receive lower values due to the absence of progress rewards and the terminal failure penalty. 3.3 Offline Goal-conditioned Policy Optimization In this work, we seek a lightweight policy improvement objective that can be seamlessly integrated into the existing action modeling loss of a pretrained VLA policy. Concretely, we adopt an AWAC-style actor update [25], which casts policy improvement as advantage-weighted behavior regression. We calculate the advantage value for each action ata_t using an n-step return estimate induced by the goal-conditioned rewards: At=(∑i=0n−1γirt+i+γnVψ(ot+n,l))−Vψ(ot,l).A_t= ( _i=0^n-1γ^ir_t+i+γ^nV_ψ(o_t+n,l) )-V_ψ(o_t,l). (8) A positive advantage indicates that the observed action leads to a higher return than expected from the current state, while a small or negative advantage suggests lower contribution to future progress. Following AWAC [25], we convert the advantage into an exponential policy improvement weight: ωt=sg[min(exp(Atβ),ωmax)], _t=sg [ ( ( A_tβ ), _ ) ], (9) where β controls the sharpness of advantage weighting, ωmax _ prevents a small number of high-advantage samples from dominating the update, and sg(⋅)sg(·) denotes the stop-gradient operation. We then optimize the policy with an AWAC-style offline RL loss over the entire relabeled dataset ~ D: ℒRL(θ)=−x~t∼~[ωtlogπθ(at∣ot,l)].L_RL(θ)=-E_ x_t D [ _t _θ(a_t o_t,l) ]. (10) This objective biases policy improvement toward high-advantage actions in the relabeled dataset: actions that contribute to achieved key states receive larger weights, while actions with low progress return have limited influence on the policy update. As a result, the policy is encouraged not merely to imitate demonstrated actions, but to improve its ability to recover from intermediate states and continue progressing toward the intended goal states. Furthermore, inspired by recent post-training paradigms in large language models (LLMs) that unify supervised and reinforcement objectives within a single optimization stage[10, 23, 36], we further retain SFT in the policy objective. Specifically, SFT is applied to successful demonstrations as a lightweight action prior, preserving low-level action feasibility and maintaining the language-action alignment inherited from the pretrained VLA model. ℒSFT(θ)=−xt∼demo[logπθ(at∣ot,l)].L_SFT(θ)=-E_x_t _demo [ _θ(a_t o_t,l) ]. (11) The final policy optimization objective is: ℒtotal(θ)=λRLℒRL(θ)+λSFTℒSFT(θ)L_total(θ)= _RLL_RL(θ)+ _SFTL_SFT(θ) (12) SFT and offline RL play complementary roles in this objective. SFT provides a coarse-grained behavioral prior, anchoring the policy to the feasible action distribution of expert demonstrations and stabilizing optimization. In contrast, offline RL provides fine-grained goal-directed optimization signals through goal-conditioned process rewards and advantage-weighted policy improvement, enabling the policy to deviate from demonstration-specific action paths when necessary and favor behaviors that better advance toward task-progress key states. For notational simplicity, we represent the action-matching term in the above objectives using the likelihood-style form logπθ(at∣ot,l) _θ(a_t o_t,l). This term serves as a general action-matching objective that encourages the policy output to align with the demonstrated action ata_t. In practice, it is instantiated according to the action head of the underlying VLA model, such as action-token likelihood [17], continuous action regression [16], denoising prediction [8, 30], or flow-matching training objectives [2, 1]. 4 Experiments 4.1 Experimental Setup Benchmarks. To evaluate the effectiveness of our method, we conduct comprehensive experiments on two widely used robotic manipulation benchmarks, LIBERO [21] and MetaWorld [35]. To further assess robustness and performance under off-demonstration conditions, we design two additional evaluation settings. First, we use LIBERO-Plus [9] to test distribution shifts in camera viewpoint, robot initial state, visual noise, and object layout. Second, to evaluate dynamic recovery under execution-time disturbances, we introduce LIBERO-ReAct, a mid-execution perturbation benchmark built on LIBERO [21]. LIBERO-ReAct includes action noise, gripper-state perturbations, and end-effector position perturbations, and is designed to evaluate goal-directed recoverability during policy execution. Backbones and Baselines. OGPO is built on the π0 _0 and π0.5 _0.5 backbones. We compare with their SFT-based few-shot and full-shot post-training variants to evaluate improvements over standard supervised imitation. We further report representative VLA models, including OpenVLA [17], SmolVLA [29], UniVLA [31], and OpenVLA-OFT [16], as broader references, and include PPO-based online RL baselines from RLinf [5] for comparison with online RL post-training. Overall, this baseline set provides a comprehensive and rigorous evaluation for assessing the effectiveness of our OGPO. 4.2 Main Results Table 1: Performance comparison on the LIBERO benchmark. Policy input notation: T (third-view image), I (instruction), W (wrist image), P (proprioception). OGPO is built upon the few-shot SFT policy, with ↑ indicating performance gains over the few-shot baseline. Model Input LIBERO Spatial Object Goal Long Avg. OpenVLA[17] T+I 84.7 88.4 79.2 53.7 76.5 π0 _0-fast[27] T+W+P+I 96.4 96.8 88.6 60.2 85.5 π0 _0[2] T+W+P+I 96.8 98.8 95.8 85.2 94.2 SmolVLA[29] T+W+P+I 93.0 94.0 91.0 77.0 88.8 WorldVLA[4] T+I 85.6 89.0 82.6 59.0 79.1 NORA[13] T+I 92.2 95.4 89.4 74.6 87.9 CoT-VLA[37] T+I 87.5 91.6 87.6 69.0 81.1 UniVLA[31] T+I 96.5 96.8 95.6 92.0 95.2 MolmoAct[18] T+I 87.0 95.4 87.6 77.2 86.6 ThinkAct[12] T+I 88.3 91.4 87.1 70.9 84.4 GROOT N1[26] T+I 94.4 97.6 93.0 90.6 93.9 OpenVLA-OFT[16] T+W+P+I 96.2 98.3 96.2 90.7 95.3 π0.5 _0.5-Full[1] T+W+P+I 98.8 98.2 98.0 92.4 96.85 TGRPO[6] T+I 90.4 92.2 81.0 59.2 80.7 VLA-RL[24] T+I 90.2 91.8 82.2 59.8 81.0 SimpleVLA-RL[19] T+I 98.2 98.7 98.8 91.7 96.9 π0.5 _0.5-PPO (RLinf)[5] T+W+P+I 99.4 99.8 98.8 94.0 98.0 π0.5 _0.5-Few Shot T+W+P+I 84.6 95.4 84.6 43.9 77.1 + OGPO T+W+P+I 99.2 98.6 97.6 93.2 97.1 ↑ 14.6 ↑ 3.2 ↑ 13.0 ↑ 49.7 ↑ 20.0 Our evaluation begins with the overall performance on widely used manipulation benchmarks. Tables 1 and 2 present the results on LIBERO and MetaWorld-MT50. On LIBERO, OGPO and PPO are initialized from the same π0.5 _0.5 few-shot policy, obtained with limited downstream demonstrations. Starting from this backbone, OGPO improves the average success rate from 77.1% to 97.1%, outperforming the π0.5 _0.5 full-SFT baseline of 96.85% and achieving performance comparable to π0.5 _0.5+PPO, an online RL method requiring costly online rollout. The improvement is particularly significant on the long-horizon LIBERO-Long suite, where our method raises the success rate from 43.9% to 93.2%, also surpassing the full-SFT result of 92.4%, indicating its enhanced goal-reaching capability in long-horizon manipulation tasks. On MetaWorld-MT50, our method consistently outperforms SFT across different backbones, improving π0 _0 from 50.8% to 81.0% and approaching the online PPO-based result of 85.8%. The gains are especially large on more challenging tasks; for example, on the π0 _0 backbone, our method improves the success rate by 28.4% on Hard tasks and 53.6% on Very Hard tasks. Overall, these results validate that OGPO can effectively extract goal-reaching supervision from offline trajectories, enabling substantial policy improvement without online rollouts, particularly on long-horizon and difficult manipulation tasks. Table 2: Performance comparison on the MetaWorld-MT50 benchmark. Policy input notation: T (third-view image), P (proprioception), C (task conditioning). C denotes the task information used to distinguish MetaWorld tasks, and may be instantiated as a language instruction, task ID, or task embedding depending on the model. Model Input MetaWorld-MT50 Easy Medium Hard Very Hard Avg. Diffusion Policy[8] T+C 23.1 10.7 1.9 6.1 10.5 TinyVLA[32] T+P+C 77.6 21.5 11.4 15.8 31.6 SmolVLA[29] T+P+C 87.1 51.8 70.0 64.0 68.2 π0 _0-PPO (RLinf)[5] T+P+I 91.1 81.8 78.3 92.0 85.8 π0 _0-SFT T+P+C 77.9 51.8 53.3 20.0 50.8 π0 _0-OGPO T+P+C 89.2 79.6 81.7 73.6 81.0 ↑ 11.3 ↑ 27.8 ↑ 28.4 ↑ 53.6 ↑ 30.2 π0.5 _0.5-SFT T+P+C 68.2 37.3 41.7 28.0 43.8 π0.5 _0.5-OGPO T+P+C 82.1 55.8 71.7 59.0 67.2 ↑ 13.9 ↑ 18.5 ↑ 30.0 ↑ 31.0 ↑ 23.4 Furthermore, to evaluate the robustness of OGPO and its ability to handle off-demonstration deviations, Table 3 reports performance under selected LIBERO-Plus perturbations and our proposed LIBERO-ReAct, a mid-execution perturbation benchmark built on LIBERO [21]. Among the perturbations provided by LIBERO-Plus, we select four representative types: camera viewpoint changes, robot initial-state variations, visual noise, and object layout changes, covering scene-, observation-, and initial-state distribution shifts. To further test dynamic recovery during policy execution, LIBERO-ReAct introduces three mid-episode action-space perturbations across the four LIBERO task suites, including Spatial, Object, Goal, and Long. For each task, we randomly select a 20-step interval during the episode and apply the perturbation after the policy predicts an action and before the action is executed in the environment: • Action noise: we add uniform noise to the executed action command, with a range of [−0.1,0.1][-0.1,0.1] for arm control and [−0.5,0.5][-0.5,0.5] for gripper control. • Gripper-state perturbation: we flip the gripper command to the opposite state, forcing an open command to close and a close command to open. • End-effector position perturbation: we add uniform noise within [−0.2,0.2][-0.2,0.2] to the predicted position motion command, simulating spatial deviations of the end effector. Compared with the full-shot SFT baseline, our offline fine-tuned π0.5 _0.5 improves the overall robustness score from 76.7% to 81.8%. It achieves clear gains on LIBERO-Plus Robot-Init and Noise perturbations, and shows even larger improvements under LIBERO-ReAct disturbances. In particular, the success rate under gripper-state perturbation increases from 76.8% to 87.7%, suggesting that our method better recovers from erroneous grasping or releasing actions. These results indicate that our method improves both robustness to static distribution shifts and recovery from execution-time disturbances. Table 3: Robustness Evaluation on the LIBERO-Plus Benchmark and Our Extended Perturbation Settings. The red-highlighted columns indicate the additional mid-episode perturbations introduced by us. All results are evaluated without perturbation-specific fine-tuning. Model Perturbation Dimensions Total Camera Robot-Init Noise Layout Action Gripper End-effector π0 _0[2] 13.8 6.0 79.0 68.9 81.5 9.3 62.4 45.8 π0 _0+fast[27] 65.1 21.6 74.4 68.8 75.2 19.8 63.8 55.5 OpenVLA[17] 0.8 3.5 15.2 28.5 19.7 24.3 31.5 17.6 OpenVLA-OFT[16] 56.4 31.9 75.8 74.2 85.2 84.4 87.6 70.8 π0.5 _0.5-SFT[1] 62.8 69.8 69.5 85.4 86.6 76.8 86.0 76.7 π0.5 _0.5-OGPO 61.5 76.6 73.1 86.6 96.2 87.7 90.1 81.8 ↓ 1.3 ↑ 6.8 ↑ 3.6 ↑ 1.2 ↑ 9.6 ↑ 10.9 ↑ 4.1 ↑ 5.1 At last, to evaluate the scalability of OGPO, we conduct an additional set of experiments using π0 _0 as the backbone. As shown in Fig. 3, OGPO also achieves substantial improvements over standard SFT on LIBERO-Spatial and LIBERO-Goal under the same 3,000 training steps. Specifically, our method reaches 93.6% on LIBERO-Spatial and 92.3% on LIBERO-Goal, outperforming SFT by 13.6 and 11.1 points, respectively. These results further validate the effectiveness of OGPO, suggesting that the proposed offline objective provides more effective learning signals than pure supervised imitation and leads to more efficient policy improvement. (a) LIBERO-Spatial (b) LIBERO-Goal Figure 3: Training efficiency comparison between Ours and SFT using the π0 _0 backbone. 4.3 Ablation Study To better understand the contribution of each component, we conduct ablation studies from two perspectives. First, we examine the reward design, which provides the basis for offline advantage estimation. Second, we analyze the policy optimization objective by removing each training signal, including offline RL and SFT. Ablation on reward designs. As shown in Fig. 4(a), the proposed reward design achieves the best performance on both LIBERO-Goal and LIBERO-Spatial. Among all variants, removing RgoalR^goal leads to the largest performance drop. This indicates that subgoal-level completion signals are crucial for offline value learning: without trajectory segmentation and subgoal relabeling, the value function cannot effectively distinguish achieved progress from failed segments. Ablation on policy fine-tuning objectives. As shown in Fig. 4(b), the full objective performs best across both tasks. Removing the offline RL term clearly degrades performance, highlighting the importance of AWAC-style advantage-weighted policy improvement for learning higher-return behaviors. Meanwhile, the decline observed after removing SFT confirms its role as positive behavior regularization, which preserves feasible successful behaviors and prevents the AWAC-style update from drifting away from the pretrained action prior. (a) Ablation on reward designs. (b) Ablation on policy fine-tuning objectives. Figure 4: Ablation studies using the π0 _0 backbone trained for 3,000 steps on LIBERO-Goal and LIBERO-Spatial. 5 Conclusion and Limitations In this work, we investigate how to leverage offline data to train a robust VLA model capable of recovering toward goal states after deviating from demonstration trajectories. Specifically, we propose OGPO, an offline goal-conditioned policy optimization framework for recoverable VLA post-training. Instead of treating demonstrations as step-wise action supervision, OGPO relabels offline trajectories into key-state-centered goal-reaching segments and transforms segment completion signals into goal-conditioned process rewards. These rewards facilitate value learning and advantage-weighted policy optimization. In addition, OGPO retains SFT to provide a coarse-grained behavioral prior that anchors the policy to the feasible action distribution induced by expert demonstrations. Overall, OGPO shifts VLA post-training from trajectory-level action imitation toward key-state reachability learning, thereby improving goal-directed recoverability without requiring online interaction. Extensive experiments demonstrate the effectiveness, robustness, and recovery capability of OGPO under off-demonstration scenarios. Despite the promising results, this work still has several limitations. First, the current gripper-state-based segmentation strategy may not generalize well to tasks whose progress cannot be naturally characterized by such events. Second, as a purely offline framework, OGPO is limited by the quality of the offline dataset, which may restrict its ability to recover from severe out-of-distribution deviations beyond the demonstrated trajectories. In future work, we plan to investigate more general and robust goal-state relabeling strategies, along with data generation and augmentation methods that can improve the diversity of offline data for recoverable policy learning. References [1] K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. R. Equi, C. Finn, N. Fusai, M. Y. Galliker, D. Ghosh, L. Groom, K. Hausman, b. ichter, S. Jakubczak, T. Jones, L. Ke, D. LeBlanc, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, A. Z. Ren, L. X. Shi, L. Smith, J. T. Springenberg, K. Stachowicz, J. Tanner, Q. Vuong, H. Walke, A. Walling, H. Wang, L. Yu, and U. Zhilinsky (2025) π0.5 _0.5: A vision-language-action model with open-world generalization. In Proceedings of The 9th Conference on Robot Learning, J. Lim, S. Song, and H. Park (Eds.), Proceedings of Machine Learning Research, Vol. 305, p. 17–40. External Links: Link Cited by: §1, §2.1, §3.3, Table 1, Table 3. [2] K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, L. X. Shi, J. Tanner, Q. Vuong, A. Walling, H. Wang, and U. Zhilinsky (2026) π0 _0: A vision-language-action flow model for general robot control. External Links: 2410.24164, Link Cited by: §1, §2.1, §3.3, Table 1, Table 3. [3] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. J. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, I. Leal, K. Lee, S. Levine, Y. Lu, U. Malla, D. Manjunath, I. Mordatch, O. Nachum, C. Parada, J. Peralta, E. Perez, K. Pertsch, J. Quiambao, K. Rao, M. Ryoo, G. Salazar, P. Sanketi, K. Sayed, J. Singh, S. Sontakke, A. Stone, C. Tan, H. Tran, V. Vanhoucke, S. Vega, Q. Vuong, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich (2023) RT-1: robotics transformer for real-world control at scale. External Links: 2212.06817, Link Cited by: §1. [4] J. Cen, C. Yu, H. Yuan, Y. Jiang, S. Huang, J. Guo, X. Li, Y. Song, H. Luo, F. Wang, D. Zhao, and H. Chen (2025) WorldVLA: towards autoregressive action world model. External Links: 2506.21539, Link Cited by: Table 1. [5] K. Chen, Z. Liu, T. Zhang, Z. Guo, S. Xu, H. Lin, H. Zang, X. Li, Q. Zhang, Z. Yu, G. Fan, T. Huang, Y. Wang, and C. Yu (2026) πRL _ RL: Online rl fine-tuning for flow-based vision-language-action models. External Links: 2510.25889, Link Cited by: Appendix E, §1, §2.2, §4.1, Table 1, Table 2. [6] Z. Chen, R. Niu, H. Kong, and Q. Wang (2025) TGRPO :fine-tuning vision-language-action model via trajectory-wise group relative policy optimization. ArXiv abs/2506.08440. External Links: Link Cited by: Table 1. [7] Z. Chen, J. Yin, Y. Chen, J. Huo, P. Tian, J. Shi, Y. Hou, Y. Li, and Y. Gao (2026) DeCo: task decomposition and skill composition for zero-shot generalization in long-horizon 3d manipulation. External Links: 2505.00527, Link Cited by: §3.1. [8] C. Chi, Z. Xu, S. Feng, E. Cousineau, Y. Du, B. Burchfiel, R. Tedrake, and S. Song (2025) Diffusion policy: visuomotor policy learning via action diffusion. The International Journal of Robotics Research 44 (10-11), p. 1684–1704. Cited by: §3.3, Table 2. [9] S. Fei, S. Wang, J. Shi, Z. Dai, J. Cai, P. Qian, L. Ji, X. He, S. Zhang, Z. Fei, J. Fu, J. Gong, and X. Qiu (2025) LIBERO-plus: in-depth robustness analysis of vision-language-action models. External Links: 2510.13626, Link Cited by: §1, §4.1. [10] Y. Fu, T. Chen, J. Chai, X. Wang, S. Tu, G. Yin, W. Lin, Q. Zhang, Y. Zhu, and D. Zhao (2026) SRFT: a single-stage method with supervised and reinforcement fine-tuning for reasoning. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §3.3. [11] Y. Guo, J. Zhang, X. Chen, X. Ji, Y. Wang, Y. Hu, and J. Chen (2025) Improving vision-language-action model with online reinforcement learning. In 2025 IEEE International Conference on Robotics and Automation (ICRA), p. 15665–15672. Cited by: §1, §2.2. [12] C. Huang, Y. Wu, M. Chen, Y. F. Wang, and F. Yang (2025) ThinkAct: vision-language-action reasoning via reinforced visual latent planning. External Links: 2507.16815, Link Cited by: Table 1. [13] C. Hung, Q. Sun, P. Hong, A. Zadeh, C. Li, U. Tan, N. Majumder, and S. Poria (2025) NORA: a small open-sourced generalist vision language action model for embodied tasks. External Links: 2504.19854, Link Cited by: Table 1. [14] Z. Jiang, S. Zhou, Y. Jiang, Z. Huang, M. Wei, Y. Chen, T. Zhou, Z. Guo, H. Lin, Q. Zhang, Y. Wang, H. Li, C. Yu, and D. Zhao (2026) WoVR: world models as reliable simulators for post-training vla policies with rl. External Links: 2602.13977, Link Cited by: §2.2. [15] T. Jülg, W. Burgard, and F. Walter (2025) Refined policy distillation: from vla generalists to rl experts. 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 11677–11684. External Links: Link Cited by: §1, §2.2. [16] M. J. Kim, C. Finn, and P. Liang (2025) Fine-tuning vision-language-action models: optimizing speed and success. External Links: 2502.19645, Link Cited by: §1, §2.1, §3.3, §4.1, Table 1, Table 3. [17] M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. P. Foster, P. R. Sanketi, Q. Vuong, T. Kollar, B. Burchfiel, R. Tedrake, D. Sadigh, S. Levine, P. Liang, and C. Finn (2025) OpenVLA: an open-source vision-language-action model. In Proceedings of The 8th Conference on Robot Learning, P. Agrawal, O. Kroemer, and W. Burgard (Eds.), Proceedings of Machine Learning Research, Vol. 270, p. 2679–2713. External Links: Link Cited by: §1, §2.1, §3.3, §4.1, Table 1, Table 3. [18] J. Lee, J. Duan, H. Fang, Y. Deng, S. Liu, B. Li, B. Fang, J. Zhang, Y. R. Wang, S. Lee, W. Han, W. Pumacay, A. Wu, R. Hendrix, K. Farley, E. VanderBilt, A. Farhadi, D. Fox, and R. Krishna (2025) MolmoAct: action reasoning models that can reason in space. External Links: 2508.07917, Link Cited by: Table 1. [19] H. Li, Y. Zuo, J. Yu, Y. Zhang, Y. Zhaohui, K. Zhang, X. Zhu, Y. Zhang, T. Chen, G. Cui, D. Wang, D. Luo, Y. Fan, Y. Sun, J. Zeng, J. Pang, S. Zhang, Y. Wang, Y. Mu, B. Zhou, and N. Ding (2026) SimpleVLA-RL: scaling VLA training via reinforcement learning. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §1, §2.2, Table 1. [20] H. Li, P. Ding, R. Suo, Z. Ge, D. Zang, K. Yu, M. Sun, H. Zhang, D. Wang, and W. Su (2026) VLA-RFT: vision-language-action reinforcement fine-tuning with verified rewards in world simulators. External Links: Link Cited by: §2.2. [21] B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone (2023) Libero: benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems 36, p. 44776–44791. Cited by: §1, §4.1, §4.2. [22] J. Liu, F. Gao, B. Wei, X. Chen, Q. Liao, Y. Wu, C. Yu, and Y. Wang (2026) What can RL bring to VLA generalization? an empirical study. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §2.2. [23] M. Liu, G. Farina, and A. E. Ozdaglar (2026) UFT: unifying supervised and reinforcement fine-tuning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §3.3. [24] G. Lu, W. Guo, C. Zhang, Y. Zhou, H. Jiang, Z. Gao, Y. Tang, and Z. Wang (2025) VLA-rl: towards masterful and general robotic manipulation with scalable reinforcement learning. External Links: 2505.18719, Link Cited by: §1, §2.2, Table 1. [25] A. Nair, A. Gupta, M. Dalal, and S. Levine (2020) Awac: accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359. Cited by: §3.3, §3.3. [26] NVIDIA, :, J. Bjorck, F. Castañeda, N. Cherniadev, X. Da, R. Ding, L. ". Fan, Y. Fang, D. Fox, F. Hu, S. Huang, J. Jang, Z. Jiang, J. Kautz, K. Kundalia, L. Lao, Z. Li, Z. Lin, K. Lin, G. Liu, E. Llontop, L. Magne, A. Mandlekar, A. Narayan, S. Nasiriany, S. Reed, Y. L. Tan, G. Wang, Z. Wang, J. Wang, Q. Wang, J. Xiang, Y. Xie, Y. Xu, Z. Xu, S. Ye, Z. Yu, A. Zhang, H. Zhang, Y. Zhao, R. Zheng, and Y. Zhu (2025) GR00T n1: an open foundation model for generalist humanoid robots. External Links: 2503.14734, Link Cited by: Table 1. [27] K. Pertsch, K. Stachowicz, B. Ichter, D. Driess, S. Nair, Q. Vuong, O. Mees, C. Finn, and S. Levine (2025) FAST: efficient action tokenization for vision-language-action models. External Links: 2501.09747, Link Cited by: Table 1, Table 3. [28] S. Ross, G. Gordon, and D. Bagnell (2011) A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, G. Gordon, D. Dunson, and M. Dudík (Eds.), Proceedings of Machine Learning Research, Vol. 15, Fort Lauderdale, FL, USA, p. 627–635. External Links: Link Cited by: §1, §2.1. [29] M. Shukor, D. Aubakirova, F. Capuano, P. Kooijmans, S. Palma, A. Zouitine, M. Aractingi, C. Pascal, M. Russi, A. Marafioti, S. Alibert, M. Cord, T. Wolf, and R. Cadène (2025) SmolVLA: a vision-language-action model for affordable and efficient robotics. ArXiv abs/2506.01844. External Links: Link Cited by: §4.1, Table 1, Table 2. [30] O. M. Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, T. Kreiman, C. Xu, et al. (2024) Octo: an open-source generalist robot policy. arXiv preprint arXiv:2405.12213. Cited by: §1, §2.1, §3.3. [31] Y. Wang, X. Li, W. Wang, J. Zhang, Y. Li, Y. Chen, X. Wang, and Z. Zhang (2026) Unified vision-language-action model. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §4.1, Table 1. [32] J. Wen, Y. Zhu, J. Li, M. Zhu, Z. Tang, K. Wu, Z. Xu, N. Liu, R. Cheng, C. Shen, Y. Peng, F. Feng, and J. Tang (2024) TinyVLA: toward fast, data-efficient vision-language-action models for robotic manipulation. IEEE Robotics and Automation Letters 10, p. 3988–3995. External Links: Link Cited by: Table 2. [33] J. Xiao, Y. Yang, X. Chang, R. Chen, F. Xiong, M. Xu, W. Zheng, and Q. Zhang (2025) World-env: leveraging world model as a virtual environment for VLA post-training. External Links: Link Cited by: §2.2. [34] W. Xiao, H. Lin, A. Peng, H. Xue, T. He, Z. Luo, Y. Xie, F. Hu, L. Fan, G. Shi, and Y. Zhu (2026) Self-improving vision-language-action models with data generation via residual RL. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §1, §2.2. [35] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine (2020) Meta-world: a benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, p. 1094–1100. Cited by: §1, §4.1. [36] W. Zhang, Y. Xie, Y. Sun, Y. Chen, G. Wang, Y. Li, B. Ding, and J. Zhou (2026) On-policy RL meets off-policy experts: harmonizing supervised fine-tuning and reinforcement learning via dynamic weighting. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §3.3. [37] Q. Zhao, Y. Lu, M. J. Kim, Z. Fu, Z. Zhang, Y. Wu, Z. Li, Q. Ma, S. Han, C. Finn, A. Handa, M. Liu, D. Xiang, G. Wetzstein, and T. Lin (2025) CoT-vla: visual chain-of-thought reasoning for vision-language-action models. External Links: 2503.22020, Link Cited by: Table 1. [38] F. Zhu, Y. Zhengyang, Z. Hong, Q. Shou, X. Ma, and S. Guo (2026) WMPO: world model-based policy optimization for vision-language-action models. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §2.2. [39] Y. Zhu, W. Wang, S. Wu, Y. Shi, and J. Wang (2025) Learning semantic atomic skills for multi-task robotic manipulation. arXiv preprint arXiv:2512.18368. Cited by: §3.1. [40] B. Zitkovich, T. Yu, S. Xu, P. Xu, T. Xiao, F. Xia, J. Wu, P. Wohlhart, S. Welker, A. Wahid, Q. Vuong, V. Vanhoucke, H. Tran, R. Soricut, A. Singh, J. Singh, P. Sermanet, P. R. Sanketi, G. Salazar, M. S. Ryoo, K. Reymann, K. Rao, K. Pertsch, I. Mordatch, H. Michalewski, Y. Lu, S. Levine, L. Lee, T. E. Lee, I. Leal, Y. Kuang, D. Kalashnikov, R. Julian, N. J. Joshi, A. Irpan, B. Ichter, J. Hsu, A. Herzog, K. Hausman, K. Gopalakrishnan, C. Fu, P. Florence, C. Finn, K. A. Dubey, D. Driess, T. Ding, K. M. Choromanski, X. Chen, Y. Chebotar, J. Carbajal, N. Brown, A. Brohan, M. G. Arenas, and K. Han (2023) RT-2: vision-language-action models transfer web knowledge to robotic control. In Proceedings of The 7th Conference on Robot Learning, J. Tan, M. Toussaint, and K. Darvish (Eds.), Proceedings of Machine Learning Research, Vol. 229, p. 2165–2183. External Links: Link Cited by: §1. Appendix A Pseudo-code of OGPO Algorithm 1 summarizes the training procedure of OGPO. The relabeled goals are only used to construct process rewards during training, and are not introduced as additional inputs to the policy or value function. Algorithm 1 OGPO: Offline Goal-conditioned Policy Optimization 1: Successful demonstrations demoD_demo, failed trajectories failD_fail, pretrained VLA policy πθ _θ, value function VψV_ψ, VLM annotator 2: Optimized policy πθ _θ 3: Initialize relabeled dataset ~←∅ D← 4: for each trajectory τ∈demo∪failτ _demo _fail do 5: Segment τ into stages τ1,…,τM\τ^1,…,τ^M\ using gripper-state changes 6: for each segment τmτ^m do 7: if τ∈demoτ _demo then 8: Set the segment terminal state as goal gmg^m and label ym←1y^m← 1 9: else 10: Use the VLM annotator to infer goal gmg^m and completion label ymy^m 11: end if 12: Add goal labels to transitions in τmτ^m 13: end for 14: end for 15: for each relabeled segment τmτ^m do 16: Compute progress variable ρtm _t^m using Eq. 3 17: Annotate process rewards rtr_t using Eq. 4 18: Add reward-augmented transitions to ~ D 19: end for 20: Value learning: Train VψV_ψ on ~ D with the n-step target in Eq. 6 by minimizing ℒVL_V in Eq. 7 21: Policy optimization: 22: while not converged do 23: Sample a relabeled minibatch from ~ D 24: Compute advantages AtA_t and AWAC weights ωt _t using the trained value function VψV_ψ 25: Compute ℒRLL_RL in Eq. 10 and ℒSFTL_SFT in Eq. 11 26: Update πθ _θ by minimizing ℒtotalL_total in Eq. 12 27: end while 28: return πθ _θ Appendix B Detailed Trajectory Segmentation and Goal Relabeling This section provides the implementation details of our trajectory segmentation and goal relabeling procedure. The key idea is to exploit the gripper open-close pattern in manipulation trajectories. In many tabletop manipulation tasks, gripper state changes naturally correspond to important semantic transitions, such as approaching an object, grasping it, transporting it, and releasing it at the target location. Therefore, we use gripper state changes as temporal anchors to divide a long trajectory into several goal-level segments. After segmentation, each segment is assigned a semantic goal label and a binary completion label. B.1 Gripper-Based Trajectory Segmentation Given an offline trajectory τ=(ot,l,at)t=1T,τ=\(o_t,l,a_t)\_t=1^T, where oto_t is the observation, l is the language instruction, and ata_t is the action, we extract the gripper command from the action sequence: ct∈open,close.c_t∈\open,close\. We then identify gripper switching points as candidate segment boundaries: ℬ=t∣ct≠ct−1.B=\t c_t≠ c_t-1\. To avoid noisy boundaries caused by short-term command oscillations, we only keep a switching point if the new gripper state is maintained for a short temporal window. The final boundary set is formed by adding the beginning and the end of the trajectory: ℬ¯=1∪ℬ∪T. B=\1\ ∪\T\. The trajectory is then divided into a sequence of segments: τ=τ1,τ2,…,τM,τm=(ot,l,at)t=smem,τ=\τ^1,τ^2,…,τ^M\, τ^m=\(o_t,l,a_t)\_t=s_m^e_m, where sms_m and eme_m are two adjacent boundaries in ℬ¯ B. This gripper-based segmentation is simple but effective for manipulation tasks because most meaningful object interactions are synchronized with gripper events. For example, an open-to-close transition near an object usually indicates a grasping attempt, while a close-to-open transition near a target region usually indicates a placing or releasing attempt. The segment between these two events often corresponds to a transportation or alignment goal. B.2 Goal Relabel After obtaining the temporal segments, we assign a semantic goal gmg^m to each segment τmτ^m. The goal label is determined by the task instruction, the gripper transition type, and the visual state change within the segment. For example, for a task instruction such as “put the black bowl in the bottom drawer of the cabinet and close it”, the trajectory can be decomposed into the following goals: reach →grasp→move→lay→close.reach . In practice, we use the following rules to assign goal labels: • If the segment ends with an open-to-close gripper transition near a target object, it is labeled as a grasping-related goal. • If the gripper remains closed and the object pose changes with the end-effector, the segment is labeled as a moving or transporting goal. • If the segment ends with a close-to-open gripper transition near the target region, it is labeled as a placing or releasing goal. • For articulated-object tasks, if the gripper remains closed while the joint state or object part pose changes, the segment is labeled as an opening, pulling, pushing, or closing goal. This procedure converts each original trajectory into a sequence of goal-conditioned segments: τ→(τ1,g1),(τ2,g2),…,(τM,gM).τ→\(τ^1,g^1),(τ^2,g^2),…,(τ^M,g^M)\. B.3 Completion Labeling For each segment τmτ^m, we further assign a binary completion label: ym∈0,1,y^m∈\0,1\, where ym=1y^m=1 indicates that the segment successfully completes its assigned goal gmg^m, and ym=0y^m=0 indicates that the segment fails to complete the goal. Successful trajectories. For a trajectory that successfully completes the full task, we label all its valid goal segments as successful: ym=1,∀m.y^m=1, ∀ m. Failed trajectories. To better localize the failure stage and provide more accurate supervision for value learning, we use a VLM-based offline annotator to determine whether each segment in a failed trajectory completes its assigned goal. The VLM is asked to judge whether the visual state at the end of the segment satisfies the assigned goal. The resulting relabeled dataset is: ~=(ot,l,at,gt,yt), D=\(o_t,l,a_t,g_t,y_t)\, where gtg_t and yty_t are inherited from the segment containing time step t. B.4 VLM Annotation Prompt for Failed Segments For failed trajectories, we use Qwen2.5-VL-72B-Instruct as the VLM annotator to evaluate whether each segment completes its candidate goal. As shown in Figure 5, the prompt provides the task instruction, candidate goal, and representative segment observations, and asks the annotator to output a completion label with a short explanation. VLM Annotation Prompt Your task is to determine whether a trajectory segment completes a given candidate subgoal. Task instruction: instruction Candidate goal: goal Segment observation: The following images show the first, middle, and last frames of a trajectory segment. Question: Does this segment successfully complete the candidate goal? Please answer with one of the following labels: SUCCESS: the final frame satisfies the candidate goal. FAILURE: the final frame does not satisfy the candidate goal. Also provide a short reason. Figure 5: Prompt template for VLM-based annotation of failed trajectory segments. We use Qwen2.5-VL-72B-Instruct as the VLM annotator. B.5 Annotated Cases Figure 6 and Figure 7 show two annotated examples. For visualization clarity, we only show frames sampled every 10 time steps as a coarse illustration. The first case is a successful trajectory. After gripper-based segmentation, all goal segments are labeled as achieved. The second case is a failed trajectory. Although the full trajectory fails, the early segments still complete meaningful goals, while the later segment fails to complete the assigned goal. Our VLM-based annotation preserves the achieved segments and marks the failed segment as negative experience. Figure 6: Annotated example of a successful trajectory. Figure 7: Annotated example of a failed trajectory. Appendix C Reward Design and Value Analysis This section provides additional details on the reward annotation procedure and analyzes the learned value estimates. Our reward design converts sparse goal completion labels into dense goal-conditioned learning signals. For a segment τmτ^m, we use the normalized progress variable ρtm _t^m to assign dense progress rewards to completed segments, while failed segments do not receive progressive positive rewards. The reward consists of three parts: a step penalty, a dense progress reward, and a segment-end success or failure term. Reward annotation. For each transition within a segment, the reward is computed according to Eq. 4. The step penalty discourages unnecessarily long or redundant action sequences. The progress reward provides smooth credit assignment for actions that move the system toward a completed goal. The terminal success bonus encourages completion of the assigned goal, while the terminal failure penalty prevents failed segments from being treated as useful demonstrations. In all experiments, we set λstep=0.01 _step=0.01, Rprog=1.0R^prog=1.0, Rsuccessgoal=1.0R^goal_success=1.0, and Rfailgoal=2.0R^goal_fail=2.0, unless otherwise specified. Value curve analysis. To verify whether the value function captures meaningful goal progress, we visualize the learned value estimates along representative trajectories. As shown in Fig. 8, the learned value curves exhibit clear local high-value regions along the execution trajectory. Interestingly, these high-value turning points are well aligned with semantically meaningful task-progress key states, such as approaching the target object, establishing a stable grasp, opening the drawer, or placing the object at the goal location. This suggests that the learned value function does not merely estimate final task success, but also captures intermediate progress toward the goal. By assigning higher values to states that are closer to reachable task-progress milestones, our method provides a progress-aware optimization signal for offline policy improvement, supporting key-state reachability learning from offline demonstrations. (a) Put the bowl on the plate. (b) Open the drawer and put the bowl inside. (c) Push the plate to the front of the stove. (d) Put the bowl on the stove. Figure 8: Visualization of the learned goal-conditioned value along execution trajectories. Local high-value frames are aligned with semantically meaningful task-progress key states, such as placing the bowl on the plate, opening the drawer, or moving the plate toward the stove. This qualitative evidence suggests that the learned value provides dense progress signals beyond final task success. Appendix D Hyperparameter Analysis In this section, we analyze the sensitivity of our method to the objective weights in Eq. 12. The final policy objective contains three terms: supervised fine-tuning on achieved goal segments, advantage-guided offline reinforcement fine-tuning, and negative suppression on failed goal segments. Their relative importance is controlled by λSFT _SFT, λRL _RLrespectively: ℒtotal(θ)=λSFTℒSFT(θ)+λRLℒRL(θ).L_total(θ)= _SFTL_SFT(θ)+ _RLL_RL(θ). Unless otherwise specified, we use λSFT=0.4,λRL=0.6. _SFT=0.4, _RL=0.6. Table 4: Sensitivity analysis of the objective weights. The default setting is highlighted in gray. λSFT _SFT λRL _RL LIBERO-Goal 0.1 0.9 79.6 0.2 0.8 83.2 0.3 0.7 86.8 0.4 0.6 92.3 0.5 0.5 89.2 0.6 0.4 82.5 0.7 0.3 82.8 Discussion. The tow terms play complementary roles. The SFT term provides a stable imitation anchor from achieved goal segments. The offline RL term further distinguishes actions according to their estimated long-term contribution to goal completion. Empirically, we find that using all three components yields more stable and effective policy improvement than relying on any single term alone. Appendix E Implementation Details We implement our offline fine-tuning method within the RLinf [5] framework. For fair comparison, our implementation follows the same preprocessing, action normalization, backbone initialization, and optimization setup as the corresponding SFT baseline. The value model is trained before policy fine-tuning and is only used during offline training to compute advantage weights. Table 5: Optimization hyperparameters. Hyperparameter LIBERO MetaWorld Optimizer AdamW AdamW Policy learning rate 2.5×10−52.5× 10^-5 2.5×10−52.5× 10^-5 Value learning rate 1.55×10−41.55× 10^-4 1.55×10−41.55× 10^-4 Adam β1 _1 0.9 0.9 Adam β2 _2 0.95 0.95 Adam ϵε 1.0×10−81.0× 10^-8 1.0×10−81.0× 10^-8 Weight decay 1.0×10−101.0× 10^-10 1.0×10−101.0× 10^-10 Gradient clipping 1.0 1.0 LR schedule Cosine warmup Cosine warmup Warmup steps 1,000 1,000 Policy Training Steps 30,000 10,000 LoRA Yes Yes LoRA rank 32 32 Table 6: Value model training settings. Setting LIBERO MetaWorld Value loss TD TD Value training epochs 5 5 Steps per epoch 600 1000 Input images 2 1 Value feature After VLM After VLM Pooling mode Mean token Mean token Freeze VLM Yes Yes Strict checkpoint loading Yes Yes Used at inference No No Table 7: Offline objective hyperparameters. Hyperparameter LIBERO MetaWorld AWAC temperature λ 1.0 1.0 Maximum advantage weight 20.0 20.0 Normalize advantage Yes Yes Advantage clipping 3.0 3.0 External reward model No No Online critic model No No Online rollouts No No Appendix F Computational Cost Analysis We compare the computational cost of our offline fine-tuning method with online RL-based post-training. Unlike online RL methods, our method does not require policy rollout collection during training. All reward annotation, value learning, and policy optimization are performed on the fixed offline dataset. The comparison shows that our offline fine-tuning method is substantially more computationally efficient than online RL-based training. More importantly, our method avoids online environment interaction, making it easier to scale to multiple task suites and safer for robotic manipulation settings where online rollout collection can be expensive or risky. Table 8: Computational cost comparison on LIBERO. GRPO denotes the online RL training recipe in RLinf. The GRPO cost is reported for one LIBERO task suite, while our method is reported for all four LIBERO suites. Method Online Rollouts Hardware Task Suites Wall-clock Time RLinf-GRPO Yes 8×A100 1 LIBERO suite ∼ 175 hours Ours No 4×RTX 4090 48GB 4 LIBERO suites ∼ 68 hours