Paper deep dive
Demonstration-Free Success-Probability Reward Learning for Generalist Robot Policies
Duo Wu, Haifeng Wang, Rongwei Lu, Jinghe Wang, Tianyi Xiong, Zhimin Wang, Chao Yu, Shuai Ma, Zhi Wang
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement learning (RL) enables generalist robot policies to improve through trial-and-error interaction, yet its effectiveness is fundamentally constrained by sparse task rewards. Existing general-purpose reward models typically alleviate this issue by learning task progress from expert demonstrations, but introduce a distribution mismatch with the mixed-quality rollouts encountered during policy optimization, making their estimates unreliable on suboptimal and failed behaviors from which the policy must learn. In this work, we introduce a demonstration-free reward learning paradigm where dense reward feedback can be learned directly from sparse task outcomes and policy experience. We theoretically show that terminal task outcomes implicitly define dense success-probability feedback at intermediate timesteps, which can be recursively learned through bootstrapping. Based on this insight, we introduce eVTA$_0$, which learns success probabilities from mixed-quality policy rollouts through temporal-difference-style bootstrapping, without expert demonstrations or intermediate annotations. We further introduce RL with Evolving Rewards (RLER), a closed-loop framework that adapts eVTA$_0$ using newly collected rollouts as the policy evolves. Experiments show that eVTA$_0$ provides more informative rewards than state-of-the-art reward models and achieves the best average policy performance across all LIBERO task suites under the same RL training budget, improving success rates by 5.4%-13.8% over the initial policy. In real-world manipulation, RLER further improves overall success rates by 20%-26%, with 35%-36% gains under out-of-distribution conditions. These results demonstrate the effectiveness of demonstration-free reward learning and adapting rewards as the policy evolves. Project webpage: this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2609.33653v1
- Canonical: https://arxiv.org/abs/2609.33653v1
Trouble viewing inline? Open PDF directly →
Full Text
94,939 characters extracted from source content.
Expand or collapse full text
DEMONSTRATION-FREE SUCCESS-PROBABILITY RE- WARD LEARNING FOR GENERALIST ROBOT POLICIES Duo Wu ♠ , Haifeng Wang ♠,♡ , Rongwei Lu ♠ , Jinghe Wang ♠ , Tianyi Xiong ♠ , Zhimin Wang ♠ , Chao Yu ♠ , Shuai Ma ♢ , Zhi Wang ♠ ♠ Tsinghua University ♢ Pengcheng Laboratory ♡ Yuanxing Robotics ABSTRACT Reinforcement learning (RL) enables generalist robot policies to improve through trial-and-error interaction, yet its effectiveness is fundamentally constrained by sparse task rewards. Existing general-purpose reward models typically alleviate this issue by learning task progress from expert demonstrations, but introduce a distribution mismatch with the mixed-quality rollouts encountered during policy optimization, making their estimates unreliable on suboptimal and failed behav- iors from which the policy must learn. In this work, we introduce a demonstration- free reward learning paradigm where dense reward feedback can be learned di- rectly from sparse task outcomes and policy experience. We theoretically show that terminal task outcomes implicitly define dense success-probability feedback at intermediate timesteps, which can be recursively learned through bootstrap- ping. Based on this insight, we introduce eVTA 0 , which learns success probabil- ities from mixed-quality policy rollouts through temporal-difference-style boot- strapping, without expert demonstrations or intermediate annotations. We fur- ther introduce RL with Evolving Rewards (RLER), a closed-loop framework that adapts eVTA 0 using newly collected rollouts as the policy evolves. Experiments show that eVTA 0 provides more informative rewards than state-of-the-art reward models and achieves the best average policy performance across all LIBERO task suites under the same RL training budget, improving success rates by 5.4%–13.8% over the initial policy. In real-world manipulation, RLER further improves over- all success rates by 20%–26%, with 35%–36% gains under out-of-distribution conditions. These results demonstrate the effectiveness of demonstration-free reward learning and adapting rewards as the policy evolves. Project webpage: https://duowuyms.github.io/evta0. 1INTRODUCTION Reinforcement learning (RL) (Li et al., 2025d; Lu et al., 2025; Li et al., 2025b; Luo et al., 2025) has recently shown great promise for enabling generalist robot policies, including vision-language- action models (VLAs) (Intelligence et al., 2025a; Kim et al., 2024; Li et al., 2024; Walke et al., 2023) and world-action models (WAMs) (Cen et al., 2025; Chen et al., 2026a; Wang et al., 2026a), to acquire new capabilities beyond demonstrations through trial-and-error experience. However, effective RL training remains constrained by sparse task rewards. In robotic manipulation, rewards are often available only as terminal success or failure signals (Mu et al., 2024; Rengarajan et al., 2022; Luo et al., 2025; Chen et al., 2025c; Zhou et al., 2025), providing little guidance on which intermediate behaviors contribute to task completion. This problem becomes particularly severe in long-horizon tasks that require hundreds or even thousands of actions before completion (Xu et al., 2025; Sermanet et al., 2024), making credit assignment and policy improvement difficult. Recent studies have explored general-purpose reward models that estimate task progress from in- termediate observations across diverse tasks and scenarios (Zhai et al., 2025; Shu et al., 2025; Liu et al., 2026a; Bai et al., 2025b; Lee et al., 2026), offering a promising approach to alleviating re- ward sparsity. Specifically, they follow a common paradigm of training reward models on expert demonstrations to estimate how much an action advances the task toward completion, under the assumption that task completion progresses monotonically over time (Din et al., 2025; Lee et al., 2026). Nevertheless, this paradigm introduces a distribution mismatch with the mixed-quality roll- outs encountered during RL training, which naturally contain suboptimal and failed trajectories. 1 arXiv:2609.33653v1 [cs.RO] 27 Sep 2026 As a result, their reward estimates can become unreliable on the suboptimal and failed behaviors from which the policy needs to learn. These limitations motivate a demonstration-free paradigm for reward learning. The central finding of this paper is that dense rewards can be learned directly from mixed-quality policy rollouts without expert demonstrations, opening a new paradigm for reward learning. We first establish the theoretical foundation behind this finding by showing that sparse terminal out- comes implicitly define dense success-probability feedback at intermediate timesteps. Specifically, the intermediate reward can be formulated as the probability of eventual task success conditioned on the trajectory history. Such success probabilities can be recursively learned through bootstrapping, allowing terminal supervision to be propagated backward without requiring intermediate annota- tions. Building on this principle, we design eVTA 0 1 , a reward model that observes manipulation histories and estimates intermediate success probabilities. We develop a simple temporal-difference (TD) style bootstrapping recipe to train eVTA 0 using only terminal signals from mixed-quality roll- outs. Furthermore, enabled by the demonstration-free learning ability of eVTA 0 , we introduce RL with Evolving Rewards (RLER), where newly collected policy rollouts are used to update eVTA 0 , and eVTA 0 in turn provides adapted rewards for subsequent policy optimization. In this way, reward learning can evolve with the changing behavior distribution of the policy. Through extensive experiments, we show that eVTA 0 produces reward signals that more reliably track task evolution and distinguish successful from failed trajectories than state-of-the-art reward models. When integrated into RL training on LIBERO (Fei et al., 2025), eVTA 0 achieves the best average success rate across all task suites under the same training budget, improving over the initial policy by 5.4%–13.8%. Moreover, in real-world manipulation, RLER progressively improves pol- icy performance, increasing overall success rates by 20%–26% over the initial SFT policies. The performance gains are more evident under out-of-distribution (OOD) conditions, with improvements of 35%–36% and 15%–20% over the SFT policy and fixed eVTA 0 baseline, respectively. The contributions of this paper are summarized as follows: • We theoretically reveal that terminal task outcomes implicitly define intermediate reward signals, where rewards can be modeled as conditional task success probabilities and recur- sively learned through bootstrapping. • Based on this principle, we establish a demonstration-free reward learning paradigm by developing eVTA 0 , a general-purpose reward model trained through a simple TD-style bootstrapping recipe to estimate success probabilities from manipulation histories. • We further introduce RL with Evolving Rewards (RLER), a closed-loop learning frame- work that adapts eVTA 0 using newly collected policy rollouts, allowing reward feedback to evolve with the changing behavior distribution of the policy. • We conduct extensive evaluations on both reward modeling and policy learning. eVTA 0 consistently provides high-quality reward feedback and improves policy performance under the same RL training budget. Real-world experiments further show that RLER improves overall success rates by 20%–26% over the initial SFT policies, while achieving 35%–36% improvements under OOD conditions. 2RELATED WORK Sparse terminal rewards hinder effective RL for training generalist robot policies, motivating a broad range of reward densification methods that can be grouped into four categories. Handcrafted dense rewards. A common strategy is to manually design dense rewards based on heuristic signals (Li et al., 2025a; Ye et al., 2025; Pan et al., 2026; Team et al., 2026; Wang et al., 2025), such as sub-goal completion (Zhang et al., 2025a), end-effector states (Xu et al., 2025), or object-gripper distances (Liu et al., 2026d). Although effective in specific settings, these methods require substantial engineering effort and often fail to generalize beyond the target tasks or scenar- ios (Cobbe et al., 2020; Tan et al., 2025). 1 eVTA stands for embodied ventral tegmental area (VTA). Inspired by the role of the biological VTA in reward processing and learning, we aim to build an embodied VTA that generates informative reward signals for continual improvement of generalist robot policies. eVTA 0 represents the first step toward this vision. 2 Stage-based rewards. To reduce reward engineering overhead, another line of work exploits the stage structure of manipulation tasks to construct dense rewards (Lu et al., 2025; Mu et al., 2024; Escoriza et al., 2025; Chen et al., 2025b). For example, SARM (Chen et al., 2025b) predicts task stages using labels derived from subtask annotations. However, these approaches still require man- ually specifying task stages, defining stage indicators, or providing subtask annotations, which may need to be redesigned for new tasks or scenarios and thus limit scalability to diverse settings. Our work instead learns success-probability feedback without requiring task-specific decomposition. World-model-derived rewards. A separate line of work leverages world models trained on expert demonstrations to derive dense rewards (Xiao et al., 2025; Liu et al., 2026b;c; Zhu et al., 2025). These methods typically learn to reconstruct observations or predict future states, and then derive rewards from reconstruction errors (Li et al., 2025c) or similarity to ground-truth observations (Hung et al., 2025). The major obstacle of these methods lies in their reliance on modeling environment dynamics and high-quality demonstrations. In comparison, our work directly models success prob- abilities without learning full environment dynamics or relying on expert demonstrations. General-purpose reward models. With the growing capabilities of foundation models, recent stud- ies have started to explore their adaptation as general-purpose reward models that estimate task progress across diverse manipulation tasks (Lee et al., 2026; Tan et al., 2025; Wang et al., 2026b; Huang et al., 2026). These methods can be categorized into training-free and training-based ap- proaches. Training-free methods, such as GVL (Ma et al., 2024) and TOPReward (Chen et al., 2026b), prompt frozen vision-language models (VLMs) to infer task progress from video frames, but they tend to produce noisy or inconsistent reward signals (Zhang et al., 2025b). To improve reward quality and stability, training-based methods instead learn progress prediction from large- scale robotic data. They pretrain vision foundation models on expert demonstrations using temporal positions of observations as pseudo progress labels, assuming that task completion evolves monoton- ically over time. However, this assumption becomes restrictive during policy learning, where roll- outs naturally contain both suboptimal and failed trajectories with non-monotonic progress. Existing methods therefore introduce additional data construction and training strategies to improve robust- ness. For example, VLAC (Zhai et al., 2025) employs pair-wise difference filtering, while Robome- ter (Liang et al., 2026) introduces inter-trajectory preference comparisons to incorporate suboptimal and failed trajectories. Nevertheless, these approaches involve complex pipelines and remain de- pendent on the quality and coverage of expert demonstrations. In comparison, eVTA 0 introduces a demonstration-free reward learning paradigm that shifts reward modeling from demonstration-based progress estimation to experience-driven success-probability learning. It directly learns intermedi- ate success probabilities from mixed-quality policy rollouts using only terminal outcomes, without requiring expert demonstrations or manually constructed intermediate progress annotations. 3METHODOLOGY In this section, we introduce eVTA 0 and its integration into RL training. We first establish the theoretical foundation of success-probability feedback, then describe the design and training method of eVTA 0 , and finally introduce RLER for adapting reward learning alongside policy optimization. 3.1THEORETICAL FOUNDATION Reward definition. Let O denote the observation space. For a rollout of length T , let ω = (o 0 ,o 1 ,...,o T ) ∈ Ω, where o t ∈ O, and Ω denotes the space of all such rollouts.Let H t = (o 0 ,o 1 ,...,o t ) denote the trajectory prefix up to timestep t. A given rollout policy π and the environment induce a probability space (Ω,F, P π ), whereF is the event space and P π is the rollout distribution. LetF t := σ(H t ) denote the information revealed by this prefix, withF t ⊆F t+1 . Each rollout terminates at T and provides only a binary terminal success label S ∈0, 1, so the terminal sparse reward is r T := S. At each intermediate timestep, we define the reward as the probability of eventual task success conditioned on the information revealed by the trajectory prefix: r t := E π [S |F t ] = P π (S = 1|F t ).(1) Theorem 1 (Temporal consistency of success-probability rewards). The success probability reward defined above satisfies r t = E π [r T |F t ] for every t≤ T . Moreover, for every t < T , it satisfies the temporal consistency property: E π [r t+1 |F t ] = r t .(2) 3 Vision Language Model o o t-w+1t-w+2t Task Reward Token Instruction Mixed-Quality Policy Rollouts eVTA 0 eVTA 0 (Target) ˆ p ˆ p − TD Loss Update Update − Reward Head Success Prob. Current History Current History Next Obs. + Figure 1: Overview of eVTA 0 . The left panel illustrates the model architecture, which estimates the probability of eventual task success conditioned on the task description and recent observation history. The right panel illustrates the training procedure, where eVTA 0 is trained on mixed-quality policy rollouts through TD-style bootstrapping. The temporal consistency property above suggests that sparse terminal rewards can be propagated backward through one-step bootstrapping. The following theorem further shows that such bootstrap- ping consistently recovers the underlying success-probability reward. Theorem 2 (Population consistency of one-step bootstrapping). Consider the ideal backward re- gression defined by the terminal anchor p ⋆ T := S and, for each t < T : p ⋆ t ∈ arg min g∈L 2 (F t ) E π h g− p ⋆ t+1 2 i ,(3) where L 2 (F t ) denotes the square-integrableF t -measurable candidate predictions. Then the popu- lation minimizer is unique up to almost-sure equality and satisfies, for every t≤ T : p ⋆ t = r t = E π [S |F t ].(4) Thus, terminal supervision can be propagated backward through ideal one-step regression without changing the population success-probability target. This result provides theoretical justification for deriving intermediate success-probability feedback from sparse terminal outcomes: although only terminal signals are available, the success probability at intermediate timesteps can be recursively characterized through one-step bootstrapping. Detailed proofs of both theorems are provided in Appendix A. 3.2DESIGN Building upon the formulation in Section 3.1, we introduce eVTA 0 to approximate the probability of eventual task success conditioned on the trajectory history. Specifically, eVTA 0 approximates the ideal success-probability reward p ⋆ t in Theorem 2 with a learnable function ˆp θ parameterized by θ. Unlike existing general-purpose reward models that estimate task progress, eVTA 0 directly models eventual task success, making its prediction naturally aligned with sparse terminal binary outcomes while providing dense intermediate feedback for policy learning. The details of eVTA 0 are described below. Model architecture. To approximate the success probability in Eq. 1, eVTA 0 predicts even- tual task success from task descriptions and recent observation histories. Since maintaining the complete trajectory prefix is computationally expensive, eVTA 0 uses a sliding context window O t = o t−w+1 ,...,o t of w recent observations as the input, where w denotes the window size. The description of the target manipulation task ℓ is additionally provided to enable task-conditioned success prediction across diverse tasks. Figure 1 depicts the architecture of eVTA 0 . To infer success probabilities from visual trajectories and task objectives, eVTA 0 is built upon a VLM to exploit its visual and linguistic understanding capabilities. The input sequence is formally defined as: x t = [I,ℓ,O t ,[REWARD]],(5) 4 where I is the instruction prompt specifying the success-probability prediction task, ℓ describes the target manipulation task,O t denotes the observation history, and [REWARD] is a special token that queries the model for the success probability. Let F ψ denote the VLM backbone. We extract the hidden representation corresponding to the [REWARD] token, h t = F ψ (x t ) [REWARD] . Next, a lightweight reward prediction head g φ is applied to estimate the success probability: ˆp θ (x t ) = σ(g φ (h t )),(6) where σ(·) denotes the sigmoid function to map the prediction into [0, 1]. Training recipe. Motivated by Theorem 2, we train eVTA 0 with a simple temporal-difference (TD) style learning recipe, where a slowly updated target eVTA 0 ˆp θ − parameterized by θ − is introduced to generate bootstrapped targets for non-terminal transitions. Specifically, we construct the training datasetD from policy-generated rollouts: D =(x t ,x + t ,d t ,z t ),(7) where d t ∈ 0, 1 indicates whether timestep t is terminal and z t ∈ 0, 1 denotes the terminal success outcome. Here, x t denotes the input of eVTA 0 ˆp θ , while x + t = [I,ℓ,O + t ,[REWARD]] represents the input of the target eVTA 0 ˆp θ − , where O + t = o t−w+1 ,...,o t ,o t+1 extends the observation history with the next observation o t+1 . This additional observation approximates the one-step-ahead information required by the bootstrap update. The TD target is then computed as: y t = d t z t + (1− d t ) ˆp θ − (x + t ).(8) For terminal transitions, the target directly uses the observed sparse outcome. For non-terminal transitions, the target eVTA 0 provides a bootstrapped estimate of the future success probability, enabling terminal supervision to propagate backward along the trajectory. The parameters of eVTA 0 are optimized by minimizing the prediction error between the estimated success probability and the TD target: L eVTA (θ) = E (x t ,x + t ,d t ,z t )∼D ( ˆp θ (x t )− y t ) 2 .(9) The target eVTA 0 is updated every K steps using exponential moving average: θ − ← αθ + (1− α)θ − ,(10) where α controls the target update rate. This TD-style bootstrapping recipe enables eVTA 0 to learn dense success-probability feedback from only terminal outcomes, without expert demonstrations or intermediate progress annotations. The detailed training procedure is summarized in Algorithm 1 of Appendix B. Training data. The training data for eVTA 0 is collected by simply rolling out a robot policy in the environment, where both successful and failed trajectories are retained to form a mixed-quality dataset. Each trajectory only provides a terminal outcome signal, with z = 1 for successful trajec- tories and z = 0 for failed ones. To better utilize failed rollouts, we also adopt an augmentation strategy to assign soft terminal targets for failed trajectories. Details are provided in Appendix B. 3.3INTEGRATION INTO RL TRAINING Reward transformation. Given the success probability ˆp t = ˆp θ (x t ) predicted by eVTA 0 , a straightforward choice is to directly use ˆp t as the dense reward. However, since ˆp t ∈ [0, 1] is non- negative, directly using it as a per-step reward may unintentionally favor longer trajectories, because additional rollout timesteps continue to contribute positive rewards even when the predicted success probability does not improve. To avoid this issue, we convert the predicted success probability into a dense reward signal as: r eVTA t =−(1− ˆp t ) = ˆp t − 1.(11) Since 1− ˆp t corresponds to the predicted probability of eventual task failure, r eVTA t can be nat- urally interpreted as a failure-probability penalty. States associated with higher probabilities of eventual failure receive larger penalties, whereas the penalty approaches zero as the predicted suc- cess probability approaches one. Notably, since this transformation only introduces a constant shift, it preserves the ordering and pairwise differences of eVTA 0 predictions. With this transformation, a 5 휋 푘 Rollouts eVTA 0 휃 Rollouts 휋 푘+1 RL Update ① ② ③ ④ ⑤ 휋 푘 Rollouts Rollouts 휋 푘+1 eVTA 0 휃 ① ③ ⑤ RL Update ⑥ ⑦ eVTA 0 휃' Densify Rewards Densify Rewards Figure 2: Illustration of RL training with a fixed eVTA 0 (left) and the proposed RL with Evolving Rewards (RLER) framework (right). RLER adapts eVTA 0 using newly collected policy rollouts, enabling reward learning and policy optimization to evolve together. trained eVTA 0 can be directly integrated into standard RL training loops, including online RL (e.g., PPO (Schulman et al., 2017) and GRPO (Guo et al., 2025)) and offline RL (e.g., IQL (Kostrikov et al., 2021), RECAP (Intelligence et al., 2025a)). RL with Evolving Rewards (RLER). The ability of eVTA 0 to learn directly from policy-generated rollouts enables reward learning to adapt alongside policy optimization. We therefore introduce RLER, a closed-loop framework in which reward modeling and policy optimization evolve together. As shown in Figure 2, new rollouts generated by the policy are periodically used to update eVTA 0 . The updated eVTA 0 then provides success-probability feedback for subsequent policy optimiza- tion. As the behavior distribution of the policy changes, updating eVTA 0 allows it to capture newly emerging behaviors and provide more informative guidance for future policy updates. 4EVALUATION We conduct extensive experiments to evaluate the effectiveness of eVTA 0 for embodied policy learn- ing, with the goal of answering the following key questions. (Q1) Reward quality: How does eVTA 0 compare with existing reward models in terms of reward quality? (Q2) Policy learning: How effectively does eVTA 0 facilitate policy learning compared with existing methods? (Q3) Re- ward evolution: Can eVTA 0 adapt to an evolving policy and further improve policy optimization through RLER? More experiments and analysis are provided in Appendix D. 4.1REWARD QUALITY Setup. We conduct experiments on two representative benchmarks LIBERO (Fei et al., 2025) and MetaWorld (Yu et al., 2020), which provide diverse task suites for comprehensive evaluation. We compare eVTA 0 against state-of-the-art general-purpose reward models, including training-free methods GVL (Ma et al., 2024) and TOPReward (Chen et al., 2026b), and pretrained reward mod- els VLAC (Zhai et al., 2025) and Robometer (Liang et al., 2026). For pretrained reward models, we evaluate both the original VLAC and Robometer models and their variants fine-tuned on the downstream benchmarks. We systematically evaluate the reward quality of each method from two complementary perspectives: temporal consistency and outcome distinction. Temporal consistency is measured on expert demonstrations using VOC (Ma et al., 2024) and VROC (Zhai et al., 2025), which assess whether rewards consistently reflect progress toward and deviation from task comple- tion, respectively. Outcome distinction is evaluated on mixed-quality policy rollouts using MSE and Kendall’s τ a (Liang et al., 2026), measuring terminal success estimation and discrimination be- tween successful and failed trajectories, respectively. Additional details on the experimental setup, baselines and eVTA 0 implementation are provided in Appendix C.1. Results. Tables 1 and 2 report the performance of each method. For temporal consistency, training- free methods generally achieve higher VOC than VROC, indicating that they can better capture forward task progression than deviations from successful completion. A similar pattern is observed for pretrained reward models. Although downstream fine-tuning substantially improves their per- formance, their VROC remains much lower than VOC. This may stem from their strong reliance on expert demonstrations, which provide limited information about how policy rollouts deviate from 6 Table 1: VOC and VROC results on LIBERO and MetaWorld. The best and second-best average results of each benchmark are shown in bold and underlined, respectively. LIBERO VOC (↑ Better)VROC (↑ Better) MethodSpatialObjectGoalLongAvg.SpatialObjectGoalLongAvg. GVL (Qwen3-VL-4B)0.140.590.120.280.280.000.160.040.030.06 GVL (Qwen3-VL-8B)0.250.460.330.370.35-0.01-0.04-0.030.100.00 GVL (GPT-5)0.790.790.720.660.740.300.530.520.570.48 TOPReward (Qwen3-VL-4B)0.700.730.740.68 0.71-0.73-0.33-0.71-0.55-0.58 TOPReward (Qwen3-VL-8B)0.650.850.760.790.76-0.59-0.13-0.30-0.34-0.34 VLAC-2B-0.010.270.000.010.07-0.080.03-0.04-0.08-0.04 VLAC-2B (Fine-tuned)0.590.600.360.370.480.610.630.350.360.49 VLAC-8B-0.060.54-0.13-0.04 0.08-0.110.52-0.23-0.030.04 VLAC-8B (Fine-tuned)0.670.650.410.390.530.590.580.350.370.47 Robometer-4B0.030.01-0.04-0.04-0.010.050.080.29-0.060.09 Robometer-4B (Fine-tuned)0.970.980.970.980.980.220.670.800.770.61 eVTA 0 (Qwen3-VL-4B)0.870.900.840.770.840.730.880.740.730.77 MetaWorld VOC (↑ Better)VROC (↑ Better) MethodEasyMediumHardVery HardAvg.EasyMediumHardVery HardAvg. GVL (Qwen3-VL-4B)0.340.170.040.180.18-0.09-0.090.040.11-0.01 GVL (Qwen3-VL-8B)0.410.220.140.160.23-0.23-0.20-0.05-0.05-0.13 GVL (GPT-5)0.820.710.400.610.640.650.59-0.020.180.35 TOPReward (Qwen3-VL-4B)0.780.780.790.830.79-0.81-0.82-0.75-0.84-0.81 TOPReward (Qwen3-VL-8B)0.860.580.440.61 0.62-0.65-0.64-0.69-0.70-0.67 VLAC-2B-0.010.06-0.020.08 0.030.070.04-0.020.080.04 VLAC-2B (Fine-tuned)0.240.130.290.360.250.410.270.460.470.40 VLAC-8B0.02-0.020.07-0.010.01-0.07-0.05-0.050.08-0.02 VLAC-8B (Fine-tuned)0.410.270.360.630.420.480.240.410.570.43 Robometer-4B-0.03-0.240.230.040.00-0.06-0.010.050.00-0.01 Robometer-4B (Fine-tuned)0.900.830.620.710.770.680.070.13-0.270.15 eVTA 0 (Qwen3-VL-4B)0.790.810.820.800.800.380.520.420.430.44 t 1 = 25 t 2 = 45 t 3 = 70t 4 = 90t 5 = 95 20 t 1 40 t 2 60 t 3 80 t 4 95(t 5 ) 0.0 0.5 1.0 eVTA 0 RobometerGVL (GPT-5) t 1 = 60t 2 = 95t 3 = 110t 4 = 125t 5 = 245 50 t 1 t 2 t 3 t 4 150200 245(t 5 ) 0.0 0.5 1.0 Task: Pick up the black bowl next to the plate and place it on the plate. Success: Starting from t 3 , the robot grasps the bowl, lifts it, and places it on the plate. Failure: Although the robot successfully grasps and lifts the bowl from t 1 to t 2 , it fails to place it stably on the plate starting from t 3 and eventually tips it over at t 5 . Figure 3: Qualitative comparison on successful and failed trajectories under the same task. successful completion. Consequently, the learned rewards tend to provide less informative feed- back on failed rollouts, which may limit their ability to guide policies to learn from failures. In comparison, eVTA 0 , despite being trained solely on mixed-quality policy rollouts without expert demonstrations, performs consistently well in both metrics. It achieves the best average VROC on both LIBERO and MetaWorld, while achieving the second-best and best average VOC, respectively. More importantly, when evaluated directly on mixed-quality policy rollouts, eVTA 0 also achieves the best average MSE and Kendall’s τ a . These results show that eVTA 0 better distinguishes success- ful and failed trajectories while providing more accurate estimates of terminal success likelihood. This capability is particularly important for RL policy learning, where collected trajectories naturally contain a mixture of successful, suboptimal, and failed rollouts. Qualitative comparison. Figure 3 further visualizes the predictions of eVTA 0 , Robometer, and GVL (GPT-5) for two trajectories of different outcomes under the same manipulation task. For the successful trajectory, eVTA 0 remains conservative during early manipulation and sharply increases its predicted success probability once the bowl is reliably moved toward and placed on the plate, eventually approaching one at task completion. For the failed trajectory, eVTA 0 initially increases its prediction when the robot successfully grasps and lifts the bowl, but decreases after the placement becomes unstable and remains low when the bowl is eventually tipped over. In contrast, Robometer 7 Table 2: MSE and Kendall’s τ a results on LIBERO and MetaWorld. The best and second-best average results of each benchmark are shown in bold and underlined, respectively. LIBERO MSE (↓ Better)Kendall’s τ a (↑ Better) MethodSpatialObjectGoalLongAvg.SpatialObjectGoalLongAvg. GVL (Qwen3-VL-4B)0.660.540.390.440.510.000.730.130.000.21 GVL (Qwen3-VL-8B)0.720.760.500.430.60-0.170.000.180.020.01 GVL (GPT-5)0.640.490.400.400.480.370.600.30-0.050.31 TOPReward (Qwen3-VL-4B)0.730.760.530.560.64-0.270.620.210.250.20 TOPReward (Qwen3-VL-8B)0.730.760.530.560.64-0.130.330.340.330.22 VLAC-2B0.720.710.520.56 0.630.000.080.02-0.020.02 VLAC-2B (Fine-tuned)0.620.730.470.510.580.490.100.210.070.22 VLAC-8B0.720.720.510.550.63-0.100.100.05-0.08-0.01 VLAC-8B (Fine-tuned)0.530.700.440.510.540.510.130.280.000.23 Robometer-4B0.250.220.190.170.210.220.09-0.02-0.160.03 Robometer-4B (Fine-tuned)0.070.090.130.10 0.100.770.770.960.900.85 eVTA 0 (Qwen3-VL-4B)0.060.030.070.080.060.981.001.000.930.98 MetaWorld MSE (↓ Better)Kendall’s τ a (↑ Better) MethodEasyMediumHardVery HardAvg.EasyMediumHardVery HardAvg. GVL (Qwen3-VL-4B)0.340.480.390.470.420.31-0.120.19-0.310.02 GVL (Qwen3-VL-8B)0.460.510.390.500.470.30-0.160.43-0.260.08 GVL (GPT-5)0.220.320.260.360.290.800.020.570.150.39 TOPReward (Qwen3-VL-4B)0.630.680.600.620.630.16-0.100.450.070.14 TOPReward (Qwen3-VL-8B)0.630.680.600.620.630.730.190.820.530.57 VLAC-2B0.610.680.600.620.630.220.000.070.020.08 VLAC-2B (Fine-tuned)0.600.620.590.600.600.470.410.430.660.49 VLAC-8B0.630.680.600.620.630.020.000.070.140.06 VLAC-8B (Fine-tuned)0.610.610.560.57 0.590.380.400.470.410.42 Robometer-4B0.170.230.140.26 0.200.730.300.780.050.47 Robometer-4B (Fine-tuned)0.080.080.140.220.130.800.630.880.300.65 eVTA 0 (Qwen3-VL-4B)0.020.050.030.060.041.000.841.000.930.94 SFTBinaryHandcraftedRobometereVTA 0 Success Rate (%) 0 20 40 60 80 100 78.8% 85.0% +6.2% 87.2% +8.4% 69.2% -9.6% 92.6% +13.8% LIBERO-Spatial 93.4% 98.6% +5.2% 89.4% -4.0% 79.0% -14.4% 99.0% +5.6% LIBERO-Object 80.4% 85.2% +4.8% 78.6% -1.8% 77.2% -3.2% 85.8% +5.4% LIBERO-Goal 43.2% 41.8% -1.4% 45.4% +2.2% 29.4% -13.8% 54.0% +10.8% LIBERO-Long 74.0% 77.7% +3.7% 75.2% +1.2% 63.7% -10.3% 82.9% +8.9% Average LIBERO-Spatial Task12345678910Average SFT 10098969276527098862078.8 Binary 989610094883896100984285.0 Handcrafted 100929698947096100963087.2 Robometer 9210010080363042100941869.2 eVTA 0 100100981009874100100985892.6 LIBERO-Long Task12345678910Average SFT 328254906205626121843.2 Binary 42228906894284402241.8 Handcrafted 58863686328042340045.4 Robometer 10066987204000829.4 eVTA 0 6264907460843626341054.0 Success Rate (%)0–20>20–40>40–60>60–80>80–100 Figure 4: Policy performance with different rewards under the same RL training budget. continues to assign high scores even after the failed trajectory deviates from successful completion, while GVL produces substantially noisier predictions. These results show that eVTA 0 better tracks the actual evolution of task success throughout policy rollouts. 4.2POLICY LEARNING Setup. We evaluate whether eVTA 0 can effectively improve RL-based policy learning on LIBERO task suites, covering diverse manipulation settings and long-horizon tasks. We use π 0.5 (Intelligence et al., 2025b) as the base policy 2 and GRPO as the RL algorithm, and compare eVTA 0 with binary reward, the handcrafted reward used in π ⋆ 0.6 (Intelligence et al., 2025a), and Robometer. The SFT policy before RL is also reported as a reference. All methods are trained under the same budget of 6,400 interaction episodes with identical RL configurations, allowing us to compare policy improve- ment and sample efficiency under a controlled setting. More details are provided in Appendix C.2. Results. Figure 4 reports the policy performance after RL training with different reward signals. eVTA 0 achieves the best average success rate on all task suites, corresponding to absolute improve- 2 We use the π 0.5 checkpoint from RLinf (Yu et al., 2026) (https://huggingface.co/RLinf/ RLinf-Pi05-LIBERO-SFT), fine-tuned on LIBERO with 40 demonstrations in total across all tasks. 8 Pick up the carrot on the moving conveyor and place it on the plate Success Rate (%) 0 20 40 60 80 100 SFTeVTA₀-RLER (Round 1)eVTA₀-RLER (Round 2)Fixed eVTA₀ 96.0% 100%100% 96.0% ID 32.0% 40.0% 68.0% 48.0% OOD 64.0% 70.0% 84.0% 72.0% Total SFT eVTA₀-RLER OOD-DistanceOOD-RotationOOD-DistanceOOD-Rotation Put the shuttlecock from the table onto the shuttlecock in the plate Success Rate (%) 0 20 40 60 80 100 SFTeVTA₀-RLER (Round 1)eVTA₀-RLER (Round 2)Fixed eVTA₀ 70.0% 86.7% 90.0% 86.7% ID 45.0% 60.0% 80.0% 65.0% OOD 60.0% 76.0% 86.0% 78.0% Total SFT eVTA₀-RLER OOD-PositionOOD-PositionOOD-PositionOOD-Position ×✓ ×✓ Figure 5: Real-world success rates across RLER rounds compared with SFT and fixed eVTA 0 . ments of 13.8%, 5.6%, 5.4%, and 10.8% on LIBERO-Spatial, LIBERO-Object, LIBERO-Goal, and LIBERO-Long over the SFT policy, respectively. In particular, the advantage is more evident on more challenging, long-horizon task suite LIBERO-Long. Handcrafted rewards yield only +2.2% changes over SFT, while binary rewards and Robometer result in 1.4% and 13.8% degradations, respectively. In contrast, eVTA 0 improves the success rate by 10.8%. The task-wise results also show that eVTA 0 maintains a more balanced performance across different tasks, whereas other re- wards exhibit large performance variations across tasks. The relatively weak RL performance of Robometer is also consistent with its less favorable reward-quality results on mixed-quality rollouts, suggesting that reward quality on policy-generated data can be important for effective policy learn- ing. These results show that eVTA 0 provides consistently effective reward guidance across diverse settings and enables more sample-efficient policy learning under the same training budget. 4.3REWARD EVOLUTION Setup. We evaluate whether reward learning can adapt alongside policy optimization through RLER in real-world manipulation tasks. Starting from the SFT policy, RLER alternates between adapting eVTA 0 with newly collected rollouts and optimizing the policy using IQL. We compare against a fixed eVTA 0 trained on the same accumulated rollout data, with matched data and policy-training budgets within each task. We report both in-distribution (ID) and out-of-distribution (OOD) perfor- mance. More details are available in Appendix C.3. Results. Figure 5 shows that RLER consistently improves policy performance over successive learn- ing rounds. Across real-world tasks, RLER improves the overall success rate by 20%–26% over the initial SFT policy and outperforms the fixed eVTA 0 baseline by 8%–12%, with particularly clear gains under OOD conditions. Notably, these improvements are achieved under matched data and policy-training budgets, suggesting that the benefit comes from adapting reward feedback as the pol- icy evolves. Conceptually, RLER transforms eVTA 0 from a static evaluator into an adaptive teacher that follows the policy’s learning frontier: as the policy encounters new behaviors and failure modes, its own rollouts refine the reward feedback, allowing the learning signal to evolve with the learner. This property is especially valuable for real-world RL, where reliable reward models are difficult to obtain beforehand and early policy rollouts are often imperfect. RLER enables reward learning to start from such imperfect experience and progressively improve alongside the policy. 5CONCLUSION In this work, we explore a new paradigm for reward learning where reward feedback can be learned directly from policy rollouts rather than relying on expert demonstrations. We show that sparse ter- minal outcomes implicitly define dense success-probability feedback, and build eVTA 0 to learn such feedback directly from mixed-quality policy rollouts without expert demonstrations or intermediate labels. Furthermore, we introduce RLER, which allows reward feedback to adapt alongside policy optimization. Experiments across simulation and real-world manipulation tasks show that eVTA 0 provides more reliable reward signals and improves policy learning, while RLER further enables reward learning to evolve with the policy. We hope this work provides a simple yet effective foun- dation toward scalable reward learning for generalist robot policies, where rewards can be learned from experience and adapt as the policy evolves. 9 AI USE STATEMENT In this work, we used generative AI tools for assisting in the writing of proofs, providing feedback on research experiments, and assisting with method implementation. We have not used genera- tive AI tools for other tasks with required disclosure, and the rest of the required disclosure tasks are not applicable to this work. Additionally, we used generative AI tools for suggesting experi- mental parameters, polishing the research paper to improve readability, and creating or modifying experimental figures or tables. We have reviewed all AI-assisted work. AI-assisted proofs and ex- perimental suggestions were carefully examined by the authors, AI-assisted code was verified and tested for correctness, and AI-assisted figures and tables were manually checked against the corre- sponding experimental results. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI. ETHICS STATEMENT This work does not involve human subjects, personally identifiable information, or other ethical concerns requiring specific discussion. REPRODUCIBILITY STATEMENT We provide the methodological and experimental details necessary to facilitate reproduction of our results. The formulation of eVTA 0 , its TD-style training procedure, and its integration into RL are described in Section 3. Complete proofs of the theoretical results are provided in Appendix A. Ap- pendix B further details the design and training of eVTA 0 , including the instruction prompt, training algorithm, and dataset augmentation strategy. Appendix C describes the experimental setups, includ- ing datasets, baselines, eVTA 0 implementation, model configurations, training hyperparameters, and evaluation settings. Additional ablation studies, qualitative analyses, and detailed experimental re- sults are reported in Appendix D. Appendix E provides additional discussion and analysis of the design and properties of eVTA 0 . We also plan to release the source code, the training and evaluation data generated in this work, and the trained model checkpoints to further facilitate reproducibility. REFERENCES Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631, 2025a. Zechen Bai, Chen Gao, and Mike Zheng Shou. Evolve-vla: Test-time training from environment feedback for vision-language-action models. arXiv preprint arXiv:2512.14666, 2025b. Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. π 0 : A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164, 2024. Jun Cen, Chaohui Yu, Hangjie Yuan, Yuming Jiang, Siteng Huang, Jiayan Guo, Xin Li, Yibing Song, Hao Luo, Fan Wang, et al. Worldvla: Towards autoregressive action world model. arXiv preprint arXiv:2506.21539, 2025. Jialei Chen, Kai Wang, Kang Chen, Shuaihang Chen, Feng Gao, Wenhao Tang, Zhiyuan Li, Weilin Liu, Zhuyu Yao, Boxun Li, et al. Lawam: Latent world action models for efficient dynamics- aware robot policies. arXiv preprint arXiv:2606.15768, 2026a. Kang Chen, Zhihao Liu, Tonghe Zhang, Zhen Guo, Si Xu, Hao Lin, Hongzhi Zang, Xiang Li, Quanlu Zhang, Zhaofei Yu, et al. π RL : Online rl fine-tuning for flow-based vision-language- action models. arXiv preprint arXiv:2510.25889, 2025a. Qianzhong Chen, Justin Yu, Mac Schwager, Pieter Abbeel, Yide Shentu, and Philipp Wu. Sarm: Stage-aware reward modeling for long horizon robot manipulation.arXiv preprint arXiv:2509.25358, 2025b. 10 Shirui Chen, Cole Harrison, Ying-Chun Lee, Angela Jin Yang, Zhongzheng Ren, Lillian J Ratliff, Jiafei Duan, Dieter Fox, and Ranjay Krishna. Topreward: Token probabilities as hidden zero-shot rewards for robotics. arXiv preprint arXiv:2602.19313, 2026b. Yuhui Chen, Shuai Tian, Shugao Liu, Yingting Zhou, Haoran Li, and Dongbin Zhao.Con- rft: A reinforced fine-tuning method for vla models via consistency policy. arXiv preprint arXiv:2502.05450, 2025c. Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning, p. 2048– 2056. PMLR, 2020. Muhayy Ud Din, Waseem Akram, Lyes Saad Saoud, Jan Rosell, and Irfan Hussain. Vision language action models in robotic manipulation: A systematic review. arXiv preprint arXiv:2507.10672, 2025. Adri ` a L ́ opez Escoriza, Nicklas Hansen, Stone Tao, Tongzhou Mu, and Hao Su. Multi-stage manip- ulation with demonstration-augmented reward, policy, and world model learning. arXiv preprint arXiv:2503.01837, 2025. Senyu Fei, Siyin Wang, Junhao Shi, Zihao Dai, Jikun Cai, Pengfang Qian, Li Ji, Xinzhe He, Shiduo Zhang, Zhaoye Fei, et al. Libero-plus: In-depth robustness analysis of vision-language-action models. arXiv preprint arXiv:2510.13626, 2025. Senyu Fei, Siyin Wang, Li Ji, Ao Li, Shiduo Zhang, Liming Liu, Jinlong Hou, Jingjing Gong, Xianzhong Zhao, and Xipeng Qiu. Srpo: Self-referential policy optimization for vision-language- action models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 6718–6728, 2026. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021. Dongchi Huang, Hongyin Zhang, Bohan Hou, Siteng Huang, Zhian Su, Hang Guo, Tong Lu, Zhaofeng Xu, Jiahao Tang, Jianfei Yang, et al. Rynnvalue: Scaling robotic value foundation models with temporal distance. arXiv preprint arXiv:2608.09853, 2026. Chia-Yu Hung, Navonil Majumder, Haoyuan Deng, Liu Renhang, Yankang Ang, Amir Zadeh, Chuan Li, Dorien Herremans, Ziwei Wang, and Soujanya Poria. Nora-1.5: A vision-language- action model trained using world model-and action-based preference rewards. arXiv preprint arXiv:2511.14659, 2025. Physical Intelligence, Ali Amin, Raichelle Aniceto, Ashwin Balakrishna, Kevin Black, Ken Conley, Grace Connors, James Darpinian, Karan Dhabalia, Jared DiCarlo, et al. π∗ 0.6 : a vla that learns from experience. arXiv preprint arXiv:2511.14759, 2025a. Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, et al. π 0.5 : a vision-language-action model with open-world generalization. arXiv preprint arXiv:2504.16054, 2025b. Jonathan Kasdin, Alison Duffy, Nathan Nadler, Arnav Raha, Adrienne L Fairhall, Kimberly L Stachenfeld, and Vikram Gadagkar. Natural behaviour is learned through dopamine-mediated reinforcement. Nature, 641(8063):699–706, 2025. Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246, 2024. Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. arXiv preprint arXiv:2110.06169, 2021. 11 Tony Lee, Andrew Wagenmaker, Karl Pertsch, Percy Liang, Sergey Levine, and Chelsea Finn. Roboreward: General-purpose vision-language reward models for robotics.arXiv preprint arXiv:2601.00675, 2026. Baicheng Li, Dong Wu, Zike Yan, Xinchen Liu, Zecui Zeng, Lusong Li, and Hongbin Zha. Reflection-based task adaptation for self-improving vla. arXiv preprint arXiv:2510.12710, 2025a. Haozhan Li, Yuxin Zuo, Jiale Yu, Yuhao Zhang, Zhaohui Yang, Kaiyan Zhang, Xuekai Zhu, Yuchen Zhang, Tianxing Chen, Ganqu Cui, et al. Simplevla-rl: Scaling vla training via reinforcement learning. arXiv preprint arXiv:2509.09674, 2025b. Hengtao Li, Pengxiang Ding, Runze Suo, Yihao Wang, Zirui Ge, Dongyuan Zang, Kexian Yu, Mingyang Sun, Hongyin Zhang, Donglin Wang, et al. Vla-rft: Vision-language-action reinforce- ment fine-tuning with verified rewards in world simulators. arXiv preprint arXiv:2510.00406, 2025c. Qixiu Li, Yaobo Liang, Zeyu Wang, Lin Luo, Xi Chen, Mozheng Liao, Fangyun Wei, Yu Deng, Sicheng Xu, Yizhong Zhang, et al. Cogact: A foundational vision-language-action model for synergizing cognition and action in robotic manipulation. arXiv preprint arXiv:2411.19650, 2024. Yunfei Li, Xiao Ma, Jiafeng Xu, Yu Cui, Zhongren Cui, Zhigang Han, Liqun Huang, Tao Kong, Yuxiao Liu, Hao Niu, et al. Gr-rl: Going dexterous and precise for long-horizon robotic manipu- lation. arXiv preprint arXiv:2512.01801, 2025d. Anthony Liang, Yigit Korkmaz, Jiahui Zhang, Minyoung Hwang, Abrar Anwar, Sidhant Kaushik, Aditya Shah, Alex S Huang, Luke Zettlemoyer, Dieter Fox, et al. Robometer: Scaling general- purpose robotic reward models via trajectory comparisons. arXiv preprint arXiv:2603.02115, 2026. Changyu Liu, Yiyang Liu, Taowen Wang, Qiao Zhuang, James Chenhao Liang, Wenhao Yang, Renjing Xu, Qifan Wang, Dongfang Liu, and Cheng Han. On-the-fly vla adaptation via test-time reinforcement learning. arXiv preprint arXiv:2601.06748, 2026a. Xiaokang Liu, Zechen Bai, Hai Ci, Kevin Yuchen Ma, and Mike Zheng Shou. World-vla-loop: Closed-loop learning of video world model and vla policy. arXiv preprint arXiv:2602.06508, 2026b. Xin Liu, Yixuan Li, Yuhui Chen, Yuxing Qin, Haoran Li, and Dongbin Zhao. Dual-granularity contrastive reward via generated episodic guidance for efficient embodied rl. arXiv preprint arXiv:2602.12636, 2026c. Yuan Liu, Haoran Li, Shuai Tian, Yuxing Qin, Yuhui Chen, Yupeng Zheng, Yongzhen Huang, and Dongbin Zhao. Towards long-lived robots: Continual learning vla models via reinforcement fine- tuning. arXiv preprint arXiv:2602.10503, 2026d. Guanxing Lu, Wenkai Guo, Chubin Zhang, Yuheng Zhou, Haonan Jiang, Zifeng Gao, Yansong Tang, and Ziwei Wang. Vla-rl: Towards masterful and general robotic manipulation with scalable reinforcement learning. arXiv preprint arXiv:2505.18719, 2025. Jianlan Luo, Charles Xu, Jeffrey Wu, and Sergey Levine. Precise and dexterous robotic manipulation via human-in-the-loop reinforcement learning. Science Robotics, 10(105):eads5033, 2025. Yecheng Jason Ma, Joey Hejna, Chuyuan Fu, Dhruv Shah, Jacky Liang, Zhuo Xu, Sean Kirmani, Peng Xu, Danny Driess, Ted Xiao, et al. Vision language models are in-context value learners. In The Thirteenth International Conference on Learning Representations, 2024. Tongzhou Mu, Minghua Liu, and Hao Su. Drs: Learning reusable dense rewards for multi-stage tasks. In International Conference on Learning Representations, volume 2024, p. 8470–8482, 2024. Xu Pan, Zhenglin Wan, Xingrui Yu, Xianwei Zheng, Youkai Ke, Ming Sun, Rui Wang, Ziwei Wang, and Ivor Tsang. Sa-vla: Spatially-aware flow-matching for vision-language-action reinforcement learning. arXiv preprint arXiv:2602.00743, 2026. 12 Desik Rengarajan, Gargi Vaidya, Akshay Sarvesh, Dileep Kalathil, and Srinivas Shakkottai. Rein- forcement learning with sparse rewards using guidance from offline demonstration. arXiv preprint arXiv:2202.04628, 2022. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. Pierre Sermanet, Tianli Ding, Jeffrey Zhao, Fei Xia, Debidatta Dwibedi, Keerthana Gopalakrishnan, Christine Chan, Gabriel Dulac-Arnold, Sharath Maddineni, Nikhil J Joshi, et al. Robovqa: Mul- timodal long-horizon reasoning for robotics. In 2024 IEEE International Conference on Robotics and Automation (ICRA), p. 645–652. IEEE, 2024. Junyang Shu, Zhiwei Lin, and Yongtao Wang. Rftf: Reinforcement fine-tuning for embodied agents with temporal feedback. arXiv preprint arXiv:2505.19767, 2025. Huajie Tan, Sixiang Chen, Yijie Xu, Zixiao Wang, Yuheng Ji, Cheng Chi, Yaoxu Lyu, Zhongxia Zhao, Xiansheng Chen, Peterson Co, et al. Robo-dopamine: General process reward modeling for high-precision robotic manipulation. arXiv preprint arXiv:2512.23703, 2025. GigaBrain Team, Boyuan Wang, Chaojun Ni, Guan Huang, Guosheng Zhao, Hao Li, Jie Li, Jindi Lv, Jingyu Liu, Lv Feng, et al. Gigabrain-0.5 m*: a vla that learns from world model-based reinforcement learning. arXiv preprint arXiv:2602.12099, 2026. Homer Rich Walke, Kevin Black, Tony Z Zhao, Quan Vuong, Chongyi Zheng, Philippe Hansen- Estruch, Andre Wang He, Vivek Myers, Moo Jin Kim, Max Du, et al. Bridgedata v2: A dataset for robot learning at scale. In Conference on robot learning, p. 1723–1736. PMLR, 2023. Si-Cheng Wang, Tian-Yu Xiang, Xiao-Hu Zhou, Mei-Jiang Gui, Xiao-Liang Xie, Shi-Qi Liu, Shuang-Yi Wang, Ao-Qun Jin, and Zeng-Guang Hou. Vla model post-training via action-chunked ppo and self behavior cloning. arXiv preprint arXiv:2509.25718, 2025. Yuran Wang, Siqiao Huang, Mingleyang Li, Chenhao Zhang, Jiaqi Liang, Weiyang Jin, Yue Chen, Xuemin Chi, Donghao Zhou, Qize Yu, Yu-Kai Wang, Yuhan Rui, Shenzhe Yao, Zhen Yuan, Zhenhao Shen, Kefei Zhu, Zijie Zhu, Ning Gao, Xiaowei Chi, Guanqi He, Shanghang Zhang, Hao Dong, Lin Shao, and Hang Zhao. Openwam: An open, modular exploration towards systematic world-action model pretraining. arXiv preprint arXiv:2609.07398, 2026a. Zhihao Wang, Jianxiong Li, Yu Cui, Yuan Gao, Xianyuan Zhan, Junzhi Yu, and Xiao Ma. World value models for robotic manipulation. arXiv preprint arXiv:2606.24742, 2026b. David Williams. Probability with Martingales. Cambridge University Press, 1991. Junjin Xiao, Yandan Yang, Xinyuan Chang, Ronghan Chen, Feng Xiong, Mu Xu, Wei-Shi Zheng, and Qing Zhang. World-env: Leveraging world model as a virtual environment for vla post- training. arXiv preprint arXiv:2509.24948, 2025. Feng Xu, Guangyao Zhai, Xin Kong, Tingzhong Fu, Daniel FN Gordon, Xueli An, and Benjamin Busam. Stare-vla: Progressive stage-aware reinforcement for fine-tuning vision-language-action models. arXiv preprint arXiv:2512.05107, 2025. Angen Ye, Zeyu Zhang, Boyuan Wang, Xiaofeng Wang, Dapeng Zhang, and Zheng Zhu. Vla-r1: Enhancing reasoning in vision-language-action models. arXiv preprint arXiv:2510.01623, 2025. Chao Yu, Yuanqing Wang, Zhen Guo, Hao Lin, Si Xu, Hongzhi Zang, Quanlu Zhang, Yongji Wu, Chunyang Zhu, Junhao Hu, et al. RLinf: Flexible and efficient Large-Scale reinforcement learning via Macro-to-Micro flow transformation. In 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI 26), p. 829–846, 2026. Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, p. 1094–1100. PMLR, 2020. 13 Shaopeng Zhai, Qi Zhang, Tianyi Zhang, Fuxian Huang, Haoran Zhang, Ming Zhou, Shengzhe Zhang, Litao Liu, Sixu Lin, and Jiangmiao Pang. A vision-language-action-critic model for robotic real-world reinforcement learning. arXiv preprint arXiv:2509.15937, 2025. Hongyin Zhang, Zifeng Zhuang, Han Zhao, Pengxiang Ding, Hongchao Lu, and Donglin Wang. ReinboT: Amplifying robot visual-language manipulation with reinforcement learning. In Pro- ceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, p. 77254–77271. PMLR, 13–19 Jul 2025a. Jiahui Zhang, Yusen Luo, Abrar Anwar, Sumedh Anand Sontakke, Joseph J Lim, Jesse Thomason, Erdem Biyik, and Jesse Zhang. Rewind: Language-guided rewards teach robot policies without new demonstrations. In Conference on Robot Learning, p. 460–488. PMLR, 2025b. Zhiyuan Zhou, Andy Peng, Qiyang Li, Sergey Levine, and Aviral Kumar. Efficient online reinforce- ment learning fine-tuning need not retain offline data. In International Conference on Learning Representations, volume 2025, p. 32343–32368, 2025. Fangqi Zhu, Zhengyang Yan, Zicong Hong, Quanxin Shou, Xiao Ma, and Song Guo. Wmpo: World model-based policy optimization for vision-language-action models.arXiv preprint arXiv:2511.09515, 2025. 14 APPENDIX CONTENTS A Proofs of Theorems16 A.1 Proof of Theorem 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16 A.2 Proof of Theorem 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .16 B Additional Details of eVTA 0 Design17 C Details of Experimental Setup18 C.1 Reward Quality Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18 C.2 Policy Learning Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .20 C.3 Real-World Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .20 D Supplementary Results22 D.1 Ablation Study of eVTA 0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22 D.2 Qualitative Analysis of Temporal Reward Dynamics . . . . . . . . . . . . . . . . .23 D.3 Detailed Real-World Experiment Results . . . . . . . . . . . . . . . . . . . . . . .24 E Discussion25 E.1 Relationship between eVTA 0 and Value Functions . . . . . . . . . . . . . . . . . .25 E.2 TD Bootstrapping versus Monte Carlo Regression . . . . . . . . . . . . . . . . . .27 15 APROOFS OF THEOREMS Recall that S ∈ 0, 1 denotes the terminal success label of a rollout of length T , F t = σ(H t ) is the information revealed by the trajectory prefix up to timestep t, and r t = E π [S | F t ] is the success-probability reward of Eq. 1. We provide the detailed proofs of each theorem as below. A.1PROOF OF THEOREM 1 Theorem 1 (Temporal consistency of success-probability rewards). The success probability reward defined above satisfies r t = E π [r T |F t ] for every t≤ T . Moreover, for every t < T , it satisfies the temporal consistency property: E π [r t+1 |F t ] = r t .(2) Proof. Since r T = S and, by definition, r t = E π [S |F t ] (Eq. 1), we have E π [r T |F t ] = E π [S |F t ] = r t , which proves the first claim. For the second claim, recall the tower property of conditional expectation (Williams, 1991): ifG 1 ⊆ G 2 and X is integrable, then: E π [E π [X |G 2 ]|G 1 ] = E π [X |G 1 ]. Applying this with X = S,G 1 = F t , andG 2 = F t+1 (admissible sinceF t ⊆ F t+1 ), together with the definition of r t+1 , gives: E π [r t+1 |F t ] = E π [E π [S |F t+1 ]|F t ] = E π [S |F t ] = r t , which completes the proof of Eq. 2. A.2PROOF OF THEOREM 2 Theorem 2 (Population consistency of one-step bootstrapping). Consider the ideal backward re- gression defined by the terminal anchor p ⋆ T := S and, for each t < T : p ⋆ t ∈ arg min g∈L 2 (F t ) E π h g− p ⋆ t+1 2 i ,(3) where L 2 (F t ) denotes the square-integrableF t -measurable candidate predictions. Then the popu- lation minimizer is unique up to almost-sure equality and satisfies, for every t≤ T : p ⋆ t = r t = E π [S |F t ].(4) Thus, terminal supervision can be propagated backward through ideal one-step regression without changing the population success-probability target. Proof. Because S ∈ 0, 1, each r t = E π [S | F t ] is bounded and therefore belongs to L 2 (F t ). Hence all of the regression risks below are finite. We prove the claim by backward induction on t. Base case (t = T ). By definition, p ⋆ T = S = r T almost surely. Induction step. Fix t < T and assume the induction hypothesis that p ⋆ t+1 = r t+1 almost surely. Define: m t := E π [p ⋆ t+1 |F t ]. The variable m t is F t -measurable and lies in L 2 (F t ). For any g ∈ L 2 (F t ), since g − p ⋆ t+1 = (g− m t ) + (m t − p ⋆ t+1 ), expanding the square gives: E π h g− p ⋆ t+1 2 i = E π (g− m t ) 2 + E π (p ⋆ t+1 − m t ) 2 + 2E π (g− m t )(m t − p ⋆ t+1 ) . For the cross term, we first evaluate the conditional expectation of its second factor. By the linearity of conditional expectation, theF t -measurability of m t , and the definition of m t , E π [m t − p ⋆ t+1 |F t ] = E π [m t |F t ]− E π [p ⋆ t+1 |F t ] = m t − m t = 0. 16 Algorithm 1 Training eVTA 0 via TD-Style Bootstrapping Input: Training datasetD, eVTA 0 ˆp θ , number of training epochs E, target update interval K, target update rate α, learning rate η, batch size B Output: Trained eVTA 0 ˆp θ 1: Initialize target parameters θ − ← θ 2: Initialize training step s← 0 3: for e = 1,...,E do 4:for each mini-batchB ⊂D with|B| = B do 5:for each (x t ,x + t ,d t ,z t )∈B do 6:if d t = 1 then 7:y t ← z t ▷ Terminal outcome 8:else 9:y t ← stopgrad ˆp θ − (x + t ) ▷ TD bootstrap 10:end if 11:end for 12:Compute the TD regression loss: L eVTA (θ) = 1 |B| X (x t ,x + t ,d t ,z t )∈B ( ˆp θ (x t )− y t ) 2 13:Update eVTA 0 : θ ← θ− η∇ θ L eVTA (θ) 14:s← s + 1 15:if s mod K = 0 then 16:θ − ← αθ + (1− α)θ − ▷ Update target eVTA 0 17:end if 18:end for 19: end for 20: return ˆp θ Moreover, since g−m t isF t -measurable, conditioning the whole product onF t and pulling out the known factor g− m t (tower property; Williams, 1991) gives: E π (g− m t )(m t − p ⋆ t+1 ) = E π (g− m t )E π [m t − p ⋆ t+1 |F t ] = 0. Consequently, E π h g− p ⋆ t+1 2 i = E π (g− m t ) 2 + E π Var π (p ⋆ t+1 |F t ) . The second term does not depend on g, whereas the first term is nonnegative and equals zero if and only if g = m t almost surely. Therefore the minimizer over L 2 (F t ), unique up to almost-sure equality, is m t . Finally, the definition of m t , the induction hypothesis, and the temporal consistency of Eq. 2 give: m t = E π [p ⋆ t+1 |F t ] = E π [r t+1 |F t ] = r t . Thus p ⋆ t = r t almost surely, completing the backward induction for all t≤ T . BADDITIONAL DETAILS OF EVTA 0 DESIGN Instruction prompt. As shown in Figure 1, we use an instruction prompt to specify the role and prediction objective of eVTA 0 : You are a reward model for robotic manipulation. Given the task instruction and the ordered history of observations, predict the probability of successful task completion. Return a scalar reward in [0, 1], where 1 indicates a high probability of success and 0 indicates a high probability of failure. The multimodal inputs are provided in the following order: task prompt, history observation images, reward token. The prompt explicitly specifies the success-probability prediction task, output range, and input or- dering for eVTA 0 . 17 Training algorithm. Algorithm 1 summarizes the training algorithm of eVTA 0 . For terminal tran- sitions, the target is given directly by the observed task outcome; for non-terminal transitions, the target is the next-step success prediction from a slowly updated target eVTA 0 . In this way, sparse ter- minal supervision is progressively propagated to intermediate timesteps, implementing the recursive regression characterized in Theorem 2. Dataset augmentation. While eVTA 0 can be trained using binary terminal outcomes alone, we adopt an additional trajectory-level augmentation strategy for failed rollouts. This is because assign- ing a hard target of zero for failed trajectories can be overly conservative, as some failures may occur after the maximum episode length is reached despite exhibiting behaviors close to task completion. Therefore, following the strategy proposed in (Fei et al., 2026), we augment the training dataset by assigning soft terminal targets to failed trajectories. The key idea is to estimate the success like- lihood of failed trajectories rather than treating all failures as equally unsuccessful. Specifically, we use a pretrained world model to extract trajectory-level representations and apply DBSCAN to cluster successful trajectories into representative success centers. For each failed trajectory, its soft terminal target z ∈ [0, 1] is computed based on the Euclidean distance to the nearest success center, normalized by the mean and standard deviation of distances across failed trajectories, and mapped through a sigmoid function. This augmentation assigns higher terminal targets to failed trajectories that exhibit behaviors closer to successful executions, providing more informative supervision for training eVTA 0 . CDETAILS OF EXPERIMENTAL SETUP C.1REWARD QUALITY EVALUATION Evaluation metrics. We provide additional details on the four metrics used in the reward quality evaluation. For temporal consistency, we use VOC (Ma et al., 2024) and VROC (Zhai et al., 2025). VOC measures the consistency between reward predictions and the forward progression of a demon- stration, while VROC examines whether the predicted reward decreases when the demonstration is temporally reversed. Together, they characterize whether a reward model responds consistently as the trajectory moves toward or away from task completion. Both metrics are computed on expert demonstrations, with higher values indicating stronger temporal consistency. For outcome distinc- tion, we use MSE and Kendall’s τ a (Liang et al., 2026). MSE measures the discrepancy between predicted terminal rewards and the associated terminal success probabilities, while Kendall’s τ a measures how well the predicted rewards distinguish successful trajectories from failed ones. To- gether, these metrics assess whether a reward model can reliably distinguish successful and failed trajectories and accurately reflect terminal success likelihood. Both metrics are computed on mixed- quality rollouts. Lower MSE and higher Kendall’s τ a indicate better outcome distinction. When computing MSE, we use the final reward prediction of each trajectory and normalize model outputs to [0, 1] when necessary, following the corresponding inference method of each model. Evaluation datasets.We evaluate reward quality on two robotic manipulation benchmarks, LIBERO and MetaWorld. LIBERO consists of four task suites, LIBERO-Spatial, LIBERO-Object, LIBERO-Goal, and LIBERO-Long, each containing 10 tasks. For MetaWorld, we evaluate 30 ma- nipulation tasks and follow prior work (Chen et al., 2025a) to organize them into four difficulty levels: Easy, Medium, Hard, and Very-Hard. Each of the Easy and Medium suites contains 10 tasks, while each of the Hard and Very-Hard suites contains 5 tasks. For temporal-consistency evaluation, we use expert demonstrations provided by both benchmarks. On LIBERO, we randomly sample 10 demonstrations per task, resulting in 400 demonstrations across 40 tasks. On MetaWorld, we similarly sample 400 demonstrations across 30 tasks: 10 demon- strations per task for the Easy and Medium suites, and 20 demonstrations per task for the Hard and Very-Hard suites. For outcome-distinction evaluation, we use the π 0 (Black et al., 2024) policy fine-tuned with RLinf (Yu et al., 2026) 3 to collect online rollouts on both LIBERO and MetaWorld. We collect 400 rollouts for each benchmark as the evaluation datasets. The LIBERO set contains 3 we use the publicly released RLinf π 0 SFT checkpoints for LIBERO and MetaWorld: https:// huggingface.co/RLinf/RLinf-Pi0-LIBERO-Spatial-Object-Goal-SFT and https:// huggingface.co/RLinf/RLinf-Pi0-MetaWorld-SFT, respectively. 18 Table 3: RL and π 0.5 training configurations for the policy learning experiments on LIBERO-Spatial / LIBERO-Object / LIBERO-Goal / LIBERO-Long. ParameterConfiguration RL configuration RL algorithmGRPO Advantage estimatorGroup-relative advantages Training budget50 epochs with 6,400 interaction episodes Parallel environments16 Rollouts per epoch8 Episodes per epoch128 GRPO group size8 Maximum episode length240 / 240 / 320 / 480 steps Batch size4 Gradient accumulation steps16 Policy learning rate2×10 −6 Adam (β 1 ,β 2 ,ε)(0.9, 0.95, 10 −8 ) Weight decay0.01 Gradient clipping1.0 Discount factor γ0.99 Policy clipping (lower / upper)0.2 / 0.2 Update epochs2 Advantage normalizationEnabled Train action expert onlyEnabled Random seed42 π 0.5 configuration Initial policyRLinf-π 0.5 -LIBERO-SFT Action chunk length10 Denoising steps5 201 successful and 199 failed trajectories, while the MetaWorld set contains 230 successful and 170 failed trajectories. Fine-tuning VLAC and Robometer. Although VLAC (Zhai et al., 2025) and Robometer (Liang et al., 2026) are pretrained on large-scale demonstration datasets, we empirically find that their per- formance degrades substantially when directly applied to LIBERO and MetaWorld without down- stream adaptation. We therefore follow their original training recipes and fine-tune both models on benchmark-specific data. Specifically, we randomly sample additional 400 demonstrations from each benchmark, without overlap with the evaluation sets. In particular, Robometer primarily learns progress from expert demonstrations and additionally incorporates failed trajectories through an auxiliary trajectory-comparison objective. Thus, we follow the original paper (Liang et al., 2026) and generate failure data by injecting random perturbations into demonstration actions and executing the resulting perturbed trajectories in the environment. This procedure yields 452 additional failed trajectories for each benchmark. We fine-tune both models for 5 epochs using LoRA (Hu et al., 2021) with rank 32 and set the batch size and learning rate to 32 and 2× 10 −5 , respectively. Implementation of eVTA 0 . We use Qwen3-VL-4B-Instruct (Bai et al., 2025a) as the backbone of eVTA 0 throughout all experiments. We set the observation context window to 5 with a temporal stride of 2. Notably, the visual observations consist of both a third-person camera image and a wrist- mounted camera image, providing complementary global and egocentric views of the manipulation process. For each benchmark, we use the SFT π 0 policy released by RLinf to collect 1,600 policy rollouts for training eVTA 0 . The LIBERO dataset contains 890 successful and 710 failed trajecto- ries, while the MetaWorld dataset follows the same composition. The rollouts cover all tasks in each benchmark and are collected independently from the evaluation sets, with no trajectory overlap. We fine-tune eVTA 0 for 5 epochs using LoRA with rank 16 and configure the batch size and learning rate as 32 and 1× 10 −5 , respectively. The target network is updated every K = 16 training steps with a soft-update rate of α = 0.1. Note that unlike VLAC and Robometer, eVTA 0 does not rely on pretraining over large-scale robotic demonstration datasets for reward modeling. Instead, it learns effective reward signals directly from mixed-quality policy rollouts. 19 Table 4: π 0.5 evaluation configurations for the policy learning experiments on LIBERO-Spatial / LIBERO-Object / LIBERO-Goal / LIBERO-Long. ParameterConfiguration Number of tasks10 / 10 / 10 / 10 Evaluation rollouts per task50 Total rollouts500 / 500 / 500 / 500 Maximum episode length240 / 240 / 320 / 480 steps Action chunk length10 Denoising steps5 Random seed42 Table 5: Configurations of SFT policies in real-world experiments (Pick Carrot / Put Shuttlecock). ParameterConfiguration Training epochs5 Expert demonstrations20 / 60 Batch size4 Gradient accumulation steps2 Learning rate1×10 −5 Gradient clipping1.0 Adam (β 1 ,β 2 ,ε)(0.9, 0.95, 10 −8 ) Fine-tuning strategyFull-model fine-tuning π 0.5 training/evaluation configuration Action chunk length32 Denoising steps5 C.2POLICY LEARNING EVALUATION Following previous work (Chen et al., 2025a), we use π 0.5 as the base policy for experiments and ini- tialize it from the few-shot SFT checkpoint released by RLinf. This checkpoint is obtained by fine- tuning π 0.5 with 40 LIBERO demonstrations, corresponding to one demonstration per task across the four task suites. Such a few-shot initialization ensures a positive success rate for each task while preserving substantial room for subsequent RL-based policy improvement. Since our goal is to eval- uate how different rewards affect subsequent RL optimization, the few-shot checkpoint provides a more informative setting for distinguishing the effects of different reward signals. Therefore, all reward methods start from the same SFT checkpoint and are trained under the same RL configura- tions, ensuring that performance differences primarily reflect the effect of the reward signal. Unless otherwise specified, the four LIBERO suites use the same training and evaluation protocol. The detailed configurations are summarized in Tables 3 and 4. C.3REAL-WORLD EVALUATION Physical setup. Figure 6 shows the physical platform of our real-world experiments. The platform consists of a Franka Research 3 robotic arm equipped with a parallel gripper and two Intel RealSense D435i RGB-D cameras. One external camera provides a third-person view of the overall workspace, while the other wrist-mounted camera captures local information around the manipulation area. Task design and configurations. We design two representative manipulation tasks covering dif- ferent scenarios. The first task requires the robot to pick up a carrot from a moving conveyor and place it on the plate. By introducing continuously changing object positions, this task evaluates the policy’s adaptation ability in dynamic environments. The second task requires the robot to put a shuttlecock from the table onto the one in the plate. Since the shuttlecock is lightweight and difficult to grasp stably, small grasping deviations can cause the object to slip or shift, making this task suit- able for evaluating grasp stability and precise control. As depicted in Figure 7, to further evaluate generalization, we construct several OOD settings by varying object positions, distances, rotations, and appearances. 20 Franka Research 3 RealSense D435i Franka Research 3 RealSense D435i Conveyor Conveyor Direction Main Image Wrist Image Main Image Wrist Image Franka Research 3 Figure 6: The physical workspace used in our real-world experiments, which is equipped with a Franka Research 3 robot arm, two Intel RealSense D435i cameras and optionally a moving conveyor. ID OOD-DistanceOOD-Rotation 1 OOD-Rotation 2OOD-Rotation 3OOD-Rotation 4 OOD-Rotation 5 Pick up the carrot on the moving conveyor and place it on the plate OOD-Color OOD-Position ID Put the shuttlecock from the table onto the shuttlecock in the plate Figure 7: The ID and OOD configurations for the real-world experiments. Evaluation method. In our real-world experiments, each task is evaluated with 50 trials covering both in-distribution (ID) and out-of-distribution (OOD) scenarios. For the Pick Carrot task, the evaluation consists of 25 ID trials and 25 OOD trials. Specifically, we perform 5 trials under the OOD-Distance setting and 4 trials for each of the five OOD-Rotation settings. For the Put Shuttle- cock task, we conduct 30 ID trials and 20 OOD trials, including 10 trials under OOD-Color and 10 trials under OOD-Position. Policy training. We use π 0.5 as the base policy. For the Pick Carrot task, we collect 20 expert demonstrations for SFT, while for the Put Shuttlecock task, we collect 60 expert demonstrations. All RL experiments start from the corresponding SFT policy, ensuring that different reward learning methods share the same initial policy. For eVTA 0 -RLER, we adopt a closed-loop reward evolution procedure. In the first round, we train eVTA 0 using 50 rollouts collected from the SFT policy, and then optimize the policy with IQL based on the trained eVTA 0 . In the second round, we collect an- other 50 rollouts from the Round 1 policy, use them to further update eVTA 0 , and perform another round of policy optimization. For the fixed eVTA 0 baseline, we aggregate the 100 rollouts collected from the SFT policy and the Round 1 policy, train eVTA 0 once on this fixed dataset, and then opti- mize the policy with IQL. Notably, the fixed eVTA 0 baseline and eVTA 0 -RLER use the same total amount of rollout data and matched optimization budgets for both reward model training and policy optimization. Therefore, the key difference between the two settings lies in the training schedule: RLER periodically adapts eVTA 0 with newly collected rollouts as the policy evolves, whereas the fixed baseline trains eVTA 0 on the aggregated rollout dataset without intermediate reward model 21 Table 6: Training configurations for IQL and eVTA 0 in real-world experiments (RLER Round 1 / RLER Round 2 / Fixed eVTA 0 ). ParameterConfiguration IQL configuration Number of trajectories50 / 50 / 100 Training epochs30 Batch size4 Gradient accumulation steps4 Discount factor γ1.0 Expectile τ0.7 Advantage temperature β0.3 Maximum advantage weight20.0 Target soft-update rate0.005 Policy learning rate1×10 −6 Value learning rate1×10 −4 Critic learning rate1×10 −4 Adam (β 1 ,β 2 ,ε)(0.9, 0.95, 10 −8 ) Policy fine-tuning strategyFull-model fine-tuning eVTA 0 configuration BackboneQwen3-VL-4B-Instruct Training epochs30 Context window5 Fine-tuning strategyLoRA LoRA rank16 Learning rate1×10 −5 Weight decay0.01 Gradient clipping0.5 Batch size32 Gradient accumulation steps2 Target-update interval16 Target soft-update rate0.1 adaptation. This comparison directly evaluates the effectiveness of closed-loop reward evolution in eVTA 0 -RLER. The detailed configurations are summarized in Table 6. DSUPPLEMENTARY RESULTS D.1ABLATION STUDY OF EVTA 0 We further study two key hyperparameters of eVTA 0 on LIBERO-Spatial: the context window size w and the target-network update interval K. Figure 8 summarizes their effects on reward quality. As shown in Figure 8 (top), increasing w from 1 to 5 substantially improves VOC, VROC, and Kendall’s τ a , with w = 5 achieving the best overall trade-off across all metrics. However, further enlarging the context window leads to a clear degradation in VROC and Kendall’s τ a , together with an overall increase in MSE, even though VOC generally continues to improve. This suggests that longer histories facilitate the recognition of overall task progression, but excessively long contexts may retain earlier positive progress even after the current behavior starts to deviate. Figure 8 (bot- tom) further reports the effect of the target-network update interval K. Reward quality shows an overall degradation as K increases. In particular, K = 16 achieves the best overall performance, whereas larger update intervals generally lead to lower VOC and VROC and higher MSE. Since eVTA 0 relies on bootstrapped targets, infrequent target-network updates can make the supervision increasingly stale and slow the propagation of updated success estimates. Based on these results, we set w = 5 and K = 16 as the default configuration of eVTA 0 and use them consistently throughout all experiments, as they provide the best overall reward quality. 22 151015202530 Context window size w 0.72 0.80 0.88 VOC ↑ 151015202530 Context window size w -0.4 0.0 0.4 0.8 VROC ↑ 151015202530 Context window size w 0.00 0.04 0.08 0.12 0.16 MSE ↓ 151015202530 Context window size w 0.88 0.92 0.96 1.00 Kendall τ a ↑ 163248648096112128 Target update interval K 0.72 0.80 0.88 VOC ↑ 163248648096112128 Target update interval K 0.75 0.78 0.80 0.83 0.85 VROC ↑ 163248648096112128 Target update interval K 0.00 0.04 0.08 0.12 0.16 MSE ↓ 163248648096112128 Target update interval K 0.88 0.92 0.96 1.00 Kendall τ a ↑ Figure 8: Ablation of hyperparameters in eVTA 0 . The top panel reports the effect of the context window size w, while the bottom part shows the effect of the target-network update interval K. D.2QUALITATIVE ANALYSIS OF TEMPORAL REWARD DYNAMICS To better understand how eVTA 0 provides reward feedback within a trajectory, we visualize its predictions on more successful and failed rollouts in Figures 9 and 10. Across both successful and failed trajectories, eVTA 0 exhibits non-uniform and behavior-responsive reward dynamics. Its prediction can remain relatively stable during phases with limited impact on task success, change sharply around transitions that substantially improve the likelihood of success, and decrease when the behavior deviates from successful completion. A representative example is the third successful trajectory in Figure 9, where the robot grasps and lifts the black bowl by t 2 , transports it toward the plate from t 3 to t 4 , and completes the place- ment by t 5 . The prediction of eVTA 0 remains relatively low after the grasp and rises sharply only when later states substantially increase the likelihood of successful completion. This is because a successful grasp alone does not guarantee eventual task success and failures may still occur during subsequent transport or placement (see Figure 10). In contrast, Robometer’s predictions increase more steadily with task progress. This reflects a fundamental difference in what the two approaches model: progress-based reward models estimate how far the task has advanced, whereas eVTA 0 es- timates the likelihood of eventual success conditioned on the current history. Consequently, eVTA 0 exhibits more non-uniform temporal reward dynamics, with larger changes concentrated around transitions that more strongly affect eventual success. The third failed trajectory in Figure 10 provides a complementary example. The robot brings the bowl close to successful placement at t 2 , subsequently moves away from the target region at t 3 , par- tially recovers at t 4 , and ultimately fails to place the bowl inside the drawer at t 5 . The prediction of eVTA 0 closely reflects this non-monotonic evolution: it increases as the robot approaches success, decreases after the deviation, rises again during recovery, and finally drops as the likelihood of suc- cessful completion decreases. Such adaptive adjustment is particularly important for mixed-quality RL rollouts, where intermediate progress does not necessarily evolve monotonically toward success, as the policy may repeatedly approach and move away from successful states while attempting to complete the task. In contrast, Robometer maintains high predictions even after the trajectory devi- ates from successful completion, revealing a mismatch between its reward estimates and the actual evolution of the task. Overall, these results show that eVTA 0 produces non-uniform and behavior-responsive reward dy- namics that reflect changes in the likelihood of task success rather than simply tracking task progress. This property is particularly useful for RL, where policy rollouts may repeatedly approach and move away from successful completion. By assigning larger reward changes to transitions that substantially affect eventual success, eVTA 0 provides more informative temporal credit assignment for distinguishing beneficial behaviors, harmful deviations, and subsequent recoveries. Interest- ingly, this behavior-responsive pattern also qualitatively resembles biological reward learning, where dopamine-related teaching signals can increase or decrease as new behaviors change the expected 23 t 1 = 30t 2 = 60t 3 = 90t 4 = 110t 5 = 120 t 1 50 t 2 t 3 100 t 4 120(t 5 ) 0.0 0.5 1.0 eVTA 0 RobometerGVL (GPT-5) t 1 = 40t 2 = 80 t 3 = 115 t 4 = 120t 5 = 160 t 1 50 t 2 100 t 3 t 4 150 160(t 5 ) 0.0 0.5 1.0 t 1 = 20t 2 = 50t 3 = 80t 4 = 85 t 5 = 115 t 1 t 2 t 3 t 4 100 115(t 5 ) 0.0 0.5 1.0 Task: Pick up the black bowl on the wooden cabinet and place it on the plate. Success: The robot approaches the bowl, grasps and lifts it at t 3 , transfers it toward the plate, and completes the placement by t 5 . Task: Pick up the cream cheese and place it in the basket. Success: The robot approaches the cream cheese, grasps it by t 2 , moves it over the basket from t 3 to t 4 , and releases it successfully by t 5 . Task: Pick up the black bowl from table center and place it on the plate. Success: The robot reaches the bowl, grasps and lifts it by t 2 , transports it toward the plate from t 3 to t 4 , and places it on the plate by t 5 . Figure 9: Additional qualitative examples of reward predictions on successful trajectories. t 1 = 5 t 2 = 80t 3 = 120t 4 = 165t 5 = 325 t 1 t 2 100 t 3 t 4 200300 325(t 5 ) 0.0 0.5 1.0 t 1 = 160t 2 = 190t 3 = 220t 4 = 265t 5 = 325 100 t 1 t 2 t 3 t 4 300 325(t 5 ) 0.0 0.5 1.0 t 1 = 150t 2 = 215t 3 = 250t 4 = 305t 5 = 345 100 t 1 t 2 t 3 t 4 345(t 5 ) 0.0 0.5 1.0 Task: Pick up the alphabet soup and place it in the basket. Failure: The robot grasps and lifts the soup can at t 2 , moves it toward the basket by t 3 , but fails to place it stably by t 5 . Task: Pick up the chocolate pudding and place it in the basket. Failure: The robot nearly completes the task at t 2 , but begins moving the pudding over the basket at t 3 and ultimately carries it away between t 4 and t 5 . Task: Open the top drawer and put the bowl inside. Failure: The robot moves the bowl toward the top drawer from t 1 and nearly succeeds at t 2 , but drifts away at t 3 , appears to recover at t 4 , and ultimately fails to place it inside the drawer at t 5 . Figure 10: Additional qualitative examples of reward predictions on failed trajectories. future outcomes (Kasdin et al., 2025). This provides an intuitive perspective on why non-uniform reward feedback can facilitate policy learning. D.3DETAILED REAL-WORLD EXPERIMENT RESULTS Tables 7 and 8 report the detailed real-world evaluation results. Across both tasks, eVTA 0 -RLER achieves the best overall performance after Round 2, with more pronounced improvements under OOD conditions. On Pick Carrot, the overall success rate increases from 32/50 with SFT to 42/50, while the OOD success rate improves from 8/25 to 17/25. On Put Shuttlecock, the correspond- ing results improve from 30/50 to 43/50 overall and from 9/20 to 16/20 under OOD conditions. eVTA 0 -RLER Round 2 also consistently outperforms the fixed eVTA 0 baseline on both tasks, fur- ther demonstrating the benefit of evolving reward feedback during policy optimization. Notably, eVTA 0 -RLER enables successful task completion in OOD settings where the SFT policy fails across all evaluation trials, improving from 0/4 to 2/4 on OOD-Rotation 3 for Pick Carrot and from 0/10 to 7/10 on OOD-Position for Put Shuttlecock. These results show that eVTA 0 -RLER not only improves 24 Table 7: Detailed real-world results for “pick up the carrot on the moving conveyor and place it on the plate”. Each entry reports the number of successful trials over the total number of trials. PolicyID OOD Distance OOD Rotation 1 OOD Rotation 2 OOD Rotation 3 OOD Rotation 4 OOD Rotation 5 Average SFT24/251/53/41/40/42/41/432/50 eVTA 0 -RLER (Round 1) 25/250/54/40/43/43/40/435/50 eVTA 0 -RLER (Round 2) 25/255/54/42/42/43/41/442/50 Fixed eVTA 0 24/251/54/41/42/42/42/436/50 Table 8: Detailed real-world results for “put the shuttlecock from the table onto the shuttlecock in the plate”. Each entry reports the number of successful trials over the total number of trials. PolicyIDOOD-ColorOOD-PositionAverage SFT21/309/100/1030/50 eVTA 0 -RLER (Round 1)26/309/103/1038/50 eVTA 0 -RLER (Round 2)27/309/107/1043/50 Fixed eVTA 0 26/309/104/1039/50 overall policy performance, but also enables successful task execution under distribution shifts that the initial SFT policy may fail to handle. Figures 11 and 12 provide the real-world rollouts illustrating the policy behaviors under OOD con- ditions. For Pick Carrot, the OOD-Distance setting places the carrot near the outer range of the robot’s workspace. The SFT policy fails to reach sufficiently far to make a stable grasp, whereas the eVTA 0 -RLER policy reaches out for the target, grasps the carrot by its leaves, and subsequently completes the transfer and placement. For Put Shuttlecock, a similar pattern is observed under OOD- Position setting. When the shuttlecock is placed farther from its ID position, the SFT policy fails to reach and grasp the object, while the eVTA 0 -RLER policy adjusts its reaching motion to the shifted target position, successfully grasps the shuttlecock, and completes the placement. EDISCUSSION E.1RELATIONSHIP BETWEEN EVTA 0 AND VALUE FUNCTIONS eVTA 0 is mathematically related to a value function under specific settings. Recall from Eq. 1 that the success-probability reward is defined as r t = P π (S = 1|F t ) = E π [S |F t ]. When the task reward consists only of a terminal success signal and the discount factor is set to γ = 1, the expected return conditioned on the trajectory history becomes V π (F t ) = E π [S |F t ] = P π (S = 1|F t ), which is equivalent to the success-probability reward approximated by eVTA 0 . Despite this connection, eVTA 0 serves a different role from the critic used for value estimation in standard RL algorithms. The key distinction lies in what they provide to policy optimization. A critic estimates the return induced by a given reward function and is used internally for value or advantage estimation. In contrast, eVTA 0 learns dense success-probability feedback from policy rollouts and task outcomes, which is transformed by Eq. 11 into the reward signal used for RL. The RL algorithm may still maintain its own critic to estimate the return induced by this reward. Therefore, eVTA 0 complements rather than replaces the critic. Furthermore, they differ in how they are coupled with policy optimization. A critic is typically updated as an internal component of the RL algorithm, whereas eVTA 0 is maintained as a separate reward model. It can remain fixed throughout policy optimization or be periodically adapted with newly collected rollouts under RLER. Empirical comparison with RL critics. Given the theoretical connection above, we further com- pare eVTA 0 with the value estimation performed by standard RL critics. We choose PPO with binary 25 OOD-Rotation 3 SFT Failure × eVTA 0 -RLER Success OOD-Rotation 5 SFT Failure × eVTA 0 -RLER Success OOD-Distance SFT Failure × eVTA 0 -RLER Success Figure 11: Real-world rollouts under OOD conditions for “pick up the carrot on the moving con- veyor and place it on the plate”. OOD-Position SFT Failure × eVTA 0 -RLER Success OOD-Position SFT Failure × eVTA 0 -RLER Success OOD-Position SFT Failure × eVTA 0 -RLER Success Figure 12: Real-world rollouts under OOD conditions for “put the shuttlecock from the table onto the shuttlecock in the plate”. terminal rewards and γ = 1 as the baseline, since under this setting the learned value function has the same target quantity as the success-probability formulation of eVTA 0 . Specifically, we conduct experiments on LIBERO-Spatial and train π 0.5 with PPO for 1,280 interaction episodes. Detailed training configurations are provided in Table 9. 26 025651276810241280 Episodes 60 70 80 90 100 Success rate (%) Binary Reward (PPO) eVTA 0 (PPO) -0.250.000.250.500.751.00 Metric value VOC ↑ VROC ↑ MSE ↓ Kendall τ a ↑ -0.26 0.87 -0.26 0.73 0.30 0.06 0.51 0.98 PPO-Critic eVTA 0 Figure 13: Comparison of implicit value estimation and explicit success-probability reward learn- ing. The left panel illustrates policy learning curves of PPO with sparse binary rewards and eVTA 0 rewards. The right panel depicts the estimation quality of the learned signals from the PPO critic and eVTA 0 . Table 9: PPO training configurations for the critic comparison experiment on LIBERO-Spatial in Appendix E.1. The binary reward and eVTA 0 reward use the same settings. ParameterConfiguration Training budget10 epochs with 1,280 interaction episodes Parallel environments16 Rollouts per epoch8 Episodes per epoch128 Maximum episode length240 steps Batch size4 Gradient accumulation steps16 Policy learning rate5×10 −6 Value learning rate1×10 −4 Adam (β 1 ,β 2 ,ε)(0.9, 0.95, 10 −8 ) Weight decay0.01 Gradient clipping1.0 Discount factor γ1.0 GAE λ0.95 PPO policy clipping0.2 PPO value clipping0.2 Update epochs1 Train action expert onlyEnabled Random seed42 As shown in Figure 13 (left), PPO with binary rewards struggles to improve and eventually degrades under sparse binary feedback, whereas replacing the sparse reward with eVTA 0 enables stable and continuous policy improvement throughout training. This suggests that explicitly learning success- probability feedback provides more effective credit assignment than relying solely on the implicit value estimation of the PPO critic. We further evaluate the estimation quality of the learned signals from the PPO critic and eVTA 0 . As shown in Figure 13 (right), eVTA 0 consistently achieves better performance in VOC, VROC, MSE, and Kendall τ a . These results demonstrate that although the PPO critic and eVTA 0 are theoretically related under sparse terminal rewards, explicitly learning a standalone success-probability reward model provides a more accurate and reliable estimation of task success, thereby offering a stronger learning signal for downstream policy learning. E.2TD BOOTSTRAPPING VERSUS MONTE CARLO REGRESSION Theoretical analysis. One natural alternative to the TD-style bootstrapping adopted in eVTA 0 is Monte Carlo (MC) regression. For each intermediate timestep t < T , recall that the success- 27 Table 10: Reward-quality comparison on LIBERO between temporal-difference (TD) bootstrapping and Monte Carlo (MC) regression. The best result is shown in bold. Task suiteMethodVOC↑VROC↑MSE↓Kendall’s τ a ↑ LIBERO-Spatial TD0.870.730.060.98 MC0.240.170.050.96 LIBERO-Object TD0.900.880.031.00 MC0.350.510.030.98 LIBERO-Goal TD0.840.740.071.00 MC0.350.320.040.97 LIBERO-Long TD0.770.730.080.93 MC0.530.480.050.80 Average TD0.840.770.060.98 MC0.370.370.040.93 probability reward is defined as: r t = E π [S |F t ] = P π (S = 1|F t ), where S ∈ 0, 1 denotes the terminal task outcome. Under MC regression, the same terminal outcome is directly used as the supervision target for every intermediate trajectory prefix: y MC t := S. This is a valid estimator of the success-probability target, since: E π y MC t |F t = E π [S |F t ] = r t . Recall the ideal one-step TD target: y TD t := r t+1 = E π [S |F t+1 ]. By the temporal consistency property in Theorem 1, the ideal TD target also has the same conditional expectation as the MC target: E π y TD t |F t = E π [r t+1 |F t ] = r t . Therefore, MC regression and TD bootstrapping share the same population target. The main differ- ence between the two approaches lies in the variance of the supervision signal. Under MC regression, each trajectory prefix is supervised by a single binary realization of the un- derlying success probability. Specifically, all prefixes from a successful trajectory receive a target of one, whereas all prefixes from a failed trajectory receive a target of zero. Since S is binary, the conditional variance of the MC target is: Var π y MC t |F t = Var π (S |F t ) = r t (1− r t ). Moreover, applying the conditional law of total variance to S gives: Var π (S |F t ) = E π [Var π (S |F t+1 )|F t ] + Var π (E π [S |F t+1 ]|F t ). Using r t+1 = E π [S |F t+1 ], this becomes: Var π (S |F t ) = E π [Var π (S |F t+1 )|F t ] + Var π (r t+1 |F t ). Since E π [Var π (S |F t+1 )|F t ] is non-negative, we have: Var π y TD t |F t = Var π (r t+1 |F t )≤ Var π (S |F t ) = Var π y MC t |F t . Thus, the TD target has lower conditional variance than the MC target. The reduction in conditional target variance is: Var π (y MC t |F t )− Var π (y TD t |F t ) = E π [Var π (S |F t+1 )|F t ]≥ 0. 28 t 1 = 15 t 2 = 40 t 3 = 55 t 4 = 70t 5 = 92 t 1 20 t 2 t 3 60 t 4 80 92(t 5 ) 0.0 0.5 1.0 TD-eVTA 0 MC-eVTA 0 t 1 = 25t 2 = 70t 3 = 105t 4 = 180t 5 = 249 t 1 50 t 2 t 3 150 t 4 200 249(t 5 ) 0.0 0.5 1.0 Task: Pick up the black bowl next to the plate and place it on the plate. Success: The robot approaches the bowl from t 1 to t 3 , grasps and transfers it toward the plate at t 4 , and completes the placement by t 5 . Task: Pick up the black bowl next to the ramekin and place it on the plate. Failure: The robot never grasps the bowl throughout the trajectory. Figure 14: Qualitative comparison of eVTA 0 trained by TD bootstrapping and MC regression on a successful trajectory (top) and a failed trajectory (bottom). Takeaway. The above analysis shows that MC regression and ideal TD bootstrapping share the same population success-probability target, while the latter provides lower-variance supervision at intermediate timesteps. Intuitively, under MC regression, since each intermediate state directly in- herits the final binary outcome, a state that is close to success can still receive a target of zero if the trajectory eventually fails, and vice versa. Hence, MC regression can be noisy for learning interme- diate success probabilities from limited rollout data. In contrast, TD bootstrapping uses the next-step success probability as a soft target for the current state, providing less noisy and more temporally informative supervision than directly assigning the final binary outcome to every intermediate state. This motivates our use of TD-style bootstrapping for learning eVTA 0 from policy rollouts. Empirical comparison. To empirically validate the advantage of TD bootstrapping, we further compare it with MC regression for training eVTA 0 on LIBERO. As shown in Table 10, TD boot- strapping achieves substantially better temporal consistency, as reflected by higher VOC and VROC scores. Meanwhile, it achieves comparable outcome distinction performance with MC regression. These results indicate that using intermediate bootstrapped success probabilities provides more in- formative supervision than directly assigning terminal binary outcomes to intermediate steps. Figure 14 further visualizes reward predictions of the two variants. For the successful trajectory, MC regression assigns consistently high rewards throughout the trajectory, providing limited distinction between early uncertain states and states closer to task completion. In contrast, TD-eVTA 0 main- tains a lower reward during early stages and increases sharply when the robot successfully grasps and places the object. For the failed trajectory, MC-eVTA 0 produces relatively high predictions despite the robot failing to make a successful grasp, whereas TD-eVTA 0 maintains low predictions throughout the trajectory. These results demonstrate that TD bootstrapping provides more informa- tive intermediate success-probability feedback and better tracks task evolution. 29