Paper deep dive
RedFlow: Redirect Failure into Action-Level Corrections for Flow-matching VLA Policy
Zhengyang Yan, Junhao Li, Fangqi Zhu, Zijun Wang, Quanxin Shou, Yikun Miao, Xiaoyi Pang, Zicong Hong, Song Guo
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 88%
Last extracted: 8/1/2026, 2:12:18 AM
Summary
RedFlow is a fine-grained offline reinforcement learning framework designed to improve flow-matching Vision-Language-Action (VLA) policies for robotic manipulation. It addresses compounding errors from distribution shifts by redirecting failure experiences into action-level corrective supervision. The framework utilizes a Context-Aware Corrective Matching mechanism to identify failure-inducing actions and retrieve successful alternatives from similar contexts, and an Adaptive Redirection Objective to reinforce successful actions, suppress undesirable ones, and redirect recoverable failures. Experiments demonstrate that RedFlow outperforms state-of-the-art offline RL baselines and matches on-policy methods like PPO, GRPO, and DDPO with significantly fewer training samples.
Entities (12)
Relation Signals (11)
RedFlow → usescomponent → Context-Aware Corrective Matching
confidence 95% · RedFlow consists of two key components: (1) a Context-Aware Corrective Matching mechanism...
RedFlow → usescomponent → Adaptive Redirection Objective
confidence 95% · ...and (2) an Adaptive Redirection Objective that jointly reinforces successful actions...
RedFlow → evaluatedon → LIBERO
confidence 90% · Experiments on the LIBERO benchmark... show that RedFlow consistently outperforms
RedFlow → improves → Flow-matching VLA Policy
confidence 90% · RedFlow... redirects failure experiences into action-level corrective supervision for flow-matching VLA policies.
RedFlow → matchesperformanceof → PPO
confidence 85% · It also matches strong on-policy methods (PPO, GRPO, and DDPO)
RedFlow → matchesperformanceof → GRPO
confidence 85% · It also matches strong on-policy methods (PPO, GRPO, and DDPO)
RedFlow → matchesperformanceof → DDPO
confidence 85% · It also matches strong on-policy methods (PPO, GRPO, and DDPO)
RedFlow → outperforms → AWR
confidence 85% · RedFlow... surpasses state-of-the-art offline RL baselines (AWR, DPO) by clear margins
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Flow-matching Vision-Language-Action (VLA) policies have shown strong potential for robotic manipulation but often suffer from compounding errors caused by distribution shifts during deployment. While offline reinforcement learning (RL) provides a practical way to improve deployed policies using rollout data, existing methods either ignore failure data or exploit it only at the trajectory level, resulting in low learning efficiency and persistent errors. We propose **RedFlow**, a fine-grained offline RL framework that redirects failure experiences into action-level corrective supervision for flow-matching VLA policies. RedFlow consists of two key components: (1) a **Context-Aware Corrective Matching** mechanism that identifies failure-inducing actions and retrieves successful alternatives from similar contexts as corrective targets, and (2) an **Adaptive Redirection Objective** that jointly reinforces successful actions, suppresses undesirable ones, and redirects recoverable failures toward corrective targets. By converting both successful and failed experiences into dense supervision, RedFlow enables robust recovery learning from mixed-quality data. Experiments on the LIBERO benchmark and three real-world manipulation tasks show that RedFlow consistently outperforms state-of-the-art offline RL baselines, improving the real-world success rate from 56.7% to 74.7%. It also matches strong on-policy methods (PPO, GRPO, and DDPO) while requiring roughly an order of magnitude fewer training samples.
Tags
Links
- Source: https://arxiv.org/abs/2607.27782v1
- Canonical: https://arxiv.org/abs/2607.27782v1
Trouble viewing inline? Open PDF directly →
Full Text
83,508 characters extracted from source content.
Expand or collapse full text
RedFlow: Redirect Failure into Action-Level Corrections for Flow-matching VLA Policy Zhengyang Yan† Junhao Li† Fangqi Zhu† Zijun Wang Quanxin Shou Yikun Miao Zicong Hong Xiaoyi Pang Song Guo* The Hong Kong University of Science and Technology zhengyang.yan@connect.ust.hk, fzhuah@connect.ust.hk † contribution. *Corresponding author. Abstract Flow-matching Vision-Language-Action (VLA) policies show great potential for robotic manipulation but frequently suffer from compounding errors caused by distribution shifts during deployment. While offline Reinforcement Learning (RL) provides a practical way to mitigate this issue by learning from deployment rollouts, existing methods either ignore failure data or use it only at a coarse trajectory level, leading to low learning efficiency and persistent errors. To solve this issue, we propose RedFlow, a fine-grained offline RL framework that Redirects failure experiences into high-fidelity action-level correction signals for Flow-matching VLA policies. RedFlow has two key components: a Context-Aware Corrective Matching mechanism that identifies failure-inducing actions and retrieves successful alternatives from similar contexts as local corrective targets, and an Adaptive Redirection Objective that modulates the training signal at three complementary levels—reinforcing successful actions, suppressing undesirable ones, and redirecting recoverable failures toward corrective targets. By turning both successes and failures into dense, structured supervision, RedFlow enables the policy to learn robust recovery behaviors from mixed-quality data. On the LIBERO benchmark and three real-robot manipulation tasks, RedFlow consistently outperforms state-of-the-art offline RL baselines, lifting the real-world success rate from 56.7% to 74.7%. The trained policy further exhibits emergent recovery behaviors that the base policy fails to produce—for instance, using the opposite arm to retrieve an out-of-reach object before retrying the task. Notably, RedFlow matches the performance of strong on-policy baselines (PPO, GRPO, DDPO) while using approximately an order of magnitude fewer training samples, establishing structured failure reuse as a sample-efficient direction for the post-training of generalist VLA policies. 1 Introduction Vision-Language-Action (VLA) policies built upon flow matching [1, 9, 36] have recently emerged as a powerful paradigm for generalist robotic manipulation. Flow matching learns a velocity field that maps noise to actions, enabling policies to model multimodal action distributions and capture the diverse strategies inherent in real-world tasks. However, since these policies are normally trained via imitation learning (IL) on human demonstrations, they inherit a fundamental limitation of behavior cloning: compounding errors under distribution shift. During real-world deployment, when the robot encounters states that deviate from the training distribution, small prediction errors accumulate over successive steps and eventually drive the policy into unrecoverable failures. Reinforcement Learning (RL) [14] enables robots to bridge the gap between training and deployment via direct learning from environment interactions. While online methods [15, 19, 33] offer continuous improvement by continually collecting and learning from fresh on-policy rollouts, their high interaction costs on real robots limit scalability. Offline RL provides a sample-efficient alternative by learning from pre-collected deployment rollouts. However, existing offline approaches each leave one of two key properties unmet (Fig. 1). Classical offline RL methods [23, 24] primarily reweight or imitate successful behaviors, failing to learn from failures and discarding the rich diagnostic information they contain. Preference-based methods [4, 34] do learn from failures, but only through coarse trajectory-level comparisons that lack action-level guidance: they signal what to avoid without specifying how the policy should improve. Human-in-the-loop interventions [8, 10, 20, 31] can offer both properties, but require expert effort that does not scale. Consequently, a significant gap remains: current frameworks lack a scalable mechanism that can extract high-fidelity action-level corrections from pre-collected failure experiences without human intervention. This motivates our central question: How can we precisely redirect failure experiences into action-level corrections that go beyond mere repulsion from bad actions and provide explicit supervisory targets for policy improvement? We identify two fundamental challenges. The first one lies in the granularity mismatch between failure labels and the required policy updates. A trajectory-level failure label obscures the sparse, action-level errors that actually cause the collapse, as many intermediate actions within a failed rollout remain perfectly reasonable. Consequently, it is inherently difficult to isolate specific failure-inducing steps and extract their exact corrective targets without dense human supervision. The second challenge is how to rigorously and precisely integrate these corrective signals into the learning process. Conventional flow-matching policies treat all provided actions as valid samples to be imitated. This “uniform imitation” is problematic when dealing with mixed-quality data: the policy may inadvertently learn sub-optimal behaviors or suffer from over-correction. Avoiding such cases when leveraging corrective signals is challenging. Figure 1: RedFlow combines two properties that prior methods lack: learning from failures and action-level guidance. (a) Preference methods learn from failures but only at the trajectory level, signaling what to avoid without action-level corrections. (b) RedFlow identifies failure-inducing actions within failed rollouts and redirects them toward corrective targets derived from successful experiences, providing dense action-level supervision. To address these challenges, we propose RedFlow, a fine-grained offline RL framework that Redirects failures into action-level correction signals for Flow-matching VLA policies. RedFlow consists of two components, each targeting one of the challenges above. First, to resolve the granularity mismatch, we propose a Context-Aware Corrective Matching mechanism. It defines an execution context using the robot’s proprioceptive state and task progress signal, identifies candidate failure-inducing action points within it, and retrieves alternatives from successful experiences in similar execution contexts. These alternatives are treated as local corrective targets: they are not assumed to be one-to-one replacement actions, but provide constructive directions for redistributing probability mass away from failure-prone regions. Second, to integrate these corrective signals precisely, we introduce an Adaptive Redirection Objective that modulates the training signal at three complementary levels based on the quality of the matched guidance: it reinforces high-quality successful actions, suppresses undesirable ones, and redirects recoverable failures toward corrective targets in the policy’s velocity field. This adaptive objective avoids the uniform imitation pitfall and aligns the corrective signals with the flow-matching velocity-field parameterization. By turning both successes and failures into dense, structured supervision, RedFlow improves sample efficiency without requiring additional human demonstrations or online interactions. Our contributions are summarized as follows: (i) We propose RedFlow, a fine-grained offline RL framework that systematically transforms failure trajectories into action-level corrective learning signals for flow-matching VLA policies, enabling the policy to learn robust recovery behaviors without requiring additional human demonstrations or online interactions. (i) We introduce a dual-component precision redirection approach: a context-aware matching procedure that identifies candidate failure points and derives high-fidelity corrective targets from successful experiences, and an adaptive redirection objective that aligns these corrective signals with the policy’s velocity-field parameterization. (i) We validate RedFlow on the LIBERO benchmark and three real-robot manipulation tasks, where it surpasses state-of-the-art offline RL baselines (AWR, DPO) by clear margins and reaches comparable performance to strong online RL baselines (PPO, GRPO, DDPO) at a fraction of the rollout cost. 2 Related Work Flow-matching VLA Policy Improvement. Vision-language-action policies with diffusion or flow-matching action heads have become a powerful paradigm for generalist robotic manipulation, as they can model multimodal continuous action distributions and inherit strong priors from large-scale vision-language backbones [1, 9, 36, 11, 22, 30, 28]. However, these policies are commonly trained by imitation learning, which treats demonstrations as uniformly desirable and provides limited mechanisms for correcting deployment failures under distribution shift. RL can further improve VLA policies by optimizing behavior on encountered states, but online approaches require repeated environment interaction and are costly for real-world robots [15, 19, 33, 35, 32]. Offline methods [23, 7, 13, 6] avoids additional interaction by learning from fixed datasets, yet standard objectives often emphasize successful behaviors, neglecting failure behaviors, or suppress undesirable actions without specifying “what the policy should do”. In contrast, RedFlow targets flow-matching VLA policy post-training in a strictly offline setting, where both successful and failed rollouts are exploited for policy improvement. Failure-Aware Learning from Process Feedback. Failed trajectories provide valuable information about policy errors, but trajectory-level failure labels are too coarse to identify which intermediate actions caused the failure. To extract finer-grained signals from execution, prior work has explored various forms of reward modeling for robotic policy learning [12, 16, 29]. These methods can assess whether a trajectory or state is moving toward success, but they remain primarily evaluative and do not specify how a continuous action should be corrected. Preference-based objectives also make use of mixed-quality data by favoring successful trajectories over failed ones [23, 4, 34, 5]. However, these methods typically guide learning through rankings, weights, or likelihood modulation, rather than producing explicit action-level corrections. Human-in-the-loop correction can provide direct action-level supervision, but requires expert intervention [8, 10, 31], limiting its scalability. RedFlow addresses this gap by jointly using progress and proprioceptive state information to transform offline failure data into action-level corrective supervision. This enables recoverable failures to be constructively redirected without additional rollouts or human corrections. 3 Methodology 3.1 Preliminaries Problem Setup We formulate language-conditioned robotic manipulation as an MDP over observation space O and action space A. At each action-chunk step t, given a task instruction l∈ℒl , the agent receives an observation ot=(It,qt)o_t=(I_t,q_t), where It∈ℐI_t denotes the visual input and qt∈ℝDqq_t ^D_q denotes the proprioceptive state. A flow-matching VLA policy πθ(at∣ot,l) _θ(a_t o_t,l) predicts an action chunk at∈ℝK×Da_t ^K× D, which consists of K consecutive D-DoF commands. The robot executes this chunk before the agent receives the next action-chunk observation ot+1o_t+1. An episode yields a trajectory τ=(ot,at)t=0T−1τ=\(o_t,a_t)\_t=0^T-1 and a binary outcome label yτ∈0,1y_τ∈\0,1\. We refer to τ as a successful trajectory when yτ=1y_τ=1 and as a failed trajectory otherwise. Optimization Objective. The goal of offline post-training is to improve the deployment policy under the task return. For a language-conditioned manipulation task, the standard RL objective is J(θ)=τ∼πθ[∑t=0T−1γtr(ot,at,l)],J(θ)=E_τ _θ [ _t=0^T-1γ^tr(o_t,a_t,l) ], (1) where r(ot,at,l)r(o_t,a_t,l) denotes the task reward and may be sparse or dominated by the terminal success outcome. At the action-chunk level, local policy improvement is governed by the advantage Aπ(ot,at,l)=Qπ(ot,at,l)−Vπ(ot,l).A^π(o_t,a_t,l)=Q^π(o_t,a_t,l)-V^π(o_t,l). (2) Actions with positive advantage should be made more likely, whereas actions with negative advantage should be down-weighted or avoided, since such updates locally improve the expected return in Eq. (1). Flow Matching VLA Policy We instantiate πθ _θ as a conditional flow-matching model [17]. Starting from noise x1∼(0,)x_1 (0,I), the policy integrates a learned velocity field vθ(xn,n,ot,l)v_θ(x_n,n,o_t,l) along the flow timestep n∈[0,1]n∈[0,1] to produce the action chunk at=x0a_t=x_0, ie, by solving the ODE dxn/dn=vθ(xn,n,ot,l)dx_n/dn=v_θ(x_n,n,o_t,l) from n=1n=1 to n=0n=0. The velocity field is pretrained on a dataset of expert demonstrations expD_exp using the standard flow-matching objective ℒFM=n,x0∼exp[‖vθ(xn,n,ot,l)−un‖2],L_FM=E_n,\,x_0 _exp [\,\|v_θ(x_n,n,o_t,l)-u_n\|^2\, ], (3) where unu_n is the conditional vector field. 3.2 Context-Aware Corrective Matching Although Eq. (1) defines the desired return-maximization objective, the fixed buffer D does not provide the true chunk-level advantage in Eq. (2). The available trajectory-level outcomes are too coarse to identify which individual chunks should be reinforced or suppressed. We therefore estimate a proxy advantage for each action chunk using learned task-progress estimates and trajectory outcomes. Figure 2: RedFlow pipeline. RedFlow consists of three stages: (I) Data Collection, where successful and failed rollouts are stored in an offline buffer; (I) Context-Aware Corrective Matching, which leverages a pre-trained GRM to estimate task progress, combines progress estimates with proprioceptive states to define execution contexts, clusters similar contexts with HDBSCAN, and derives corrective targets from positive actions observed in matched contexts; and (I) Adaptive Redirection Objective, where asymmetric flow matching reinforces high-quality chunks, suppresses negative chunks, and redirects correctable failure-inducing chunks. Action-level advantage estimation. To obtain an offline surrogate for the chunk-level advantage, we convert learned task-progress estimates and trajectory outcomes into a signed score. We use a pretrained General Reward Model (GRM) [29], R(ot,l)∈[0,1]R(o_t,l)∈[0,1], as the source of task-progress estimates from the observation and instruction. Since raw GRM scores may fluctuate across nearby chunks, we smooth the progress sequence with a box filter of half-window size W>0W>0: p¯t=12W+1∑j=t−Wt+WR(oj,l), p_t= 12W+1 _j=t-W^t+WR(o_j,l), (4) with one-sided averaging at the boundaries. We then convert the smoothed progress sequence into a signed action-level score by combining local progress change with a trajectory-level outcome bias: A^t=p¯t+W−p¯t−W+b⋅(2 1[yτ=1]−1), A_t= p_t+W- p_t-W+b· (2\, 1[y_τ=1]-1 ), (5) where b>0b>0 is a fixed coefficient and yτy_τ is the outcome label of the trajectory containing ata_t. The local progress-change term captures whether the trajectory advances around chunk ata_t, while the outcome-bias term injects coarse success or failure information when local progress alone is ambiguous. We clip the indices t−Wt-W and t+Wt+W to valid trajectory boundaries. We use the sign of A^t A_t as the action-level label: chunks with A^t>0 A_t>0 are labeled positive, chunks with A^t<0 A_t<0 are labeled negative, and the zero case is handled through the soft weight in Eq. (9) rather than used for corrective-target assignment. Context clustering for corrective targets. To contextualize these action labels, we define a chunk’s execution context via the task instruction, normalized proprioceptive state, and smoothed progress estimate, and operationalize it for clustering through the progress–state feature below. The task instruction is fixed within each task-specific clustering problem, while the progress–state feature captures local execution stage and robot configuration. Specifically, we define the progress–state space using the proprioceptive state qtq_t and the smoothed progress estimate p¯t p_t. Our key observation is that chunks close in this space often correspond to the same underlying subtask, even when they come from trajectories with different final outcomes. The proprioceptive state captures the robot configuration, while the GRM-predicted progress implicitly incorporates task-relevant visual information. This design avoids direct clustering in high-dimensional visual space. We instantiate this space with the feature ft=[q~t;βp¯t],f_t=[\, q_t;\;β\, p_t\,], (6) where q~t q_t denotes normalized proprioception and β>0β>0 balances the relative scales of task progress and proprioceptive state. For each task, we cluster all action chunks in D using HDBSCAN [21] over ft\f_t\, yielding clusters c\C_c\. For a cluster cC_c, let c+=i∈c:A^i>0C_c^+=\\,i _c: A_i>0\,\ denote its positive subset. For each negative chunk ata_t with A^t<0 A_t<0 assigned to cC_c, if c+C_c^+ is non-empty, we construct a corrective target as a quality-weighted action centroid of the positive chunks in the same cluster: αi=exp(A^i/κ)∑j∈c+exp(A^j/κ),at⋆=∑i∈c+αiai, _i= ( A_i/κ) _j _c^+ ( A_j/κ), a_t = _i _c^+ _ia_i, (7) where κ>0κ>0 controls the concentration of the centroid toward high-advantage chunks. A smaller κ makes the corrective target closer to the highest-advantage positive actions, while a larger κ yields a more uniform cluster centroid. Crucially, at∗a_t^* acts not as a one-to-one counterfactual replacement, but as an empirical positive barycenter defining a local transport direction to redistribute probability mass away from failure modes. A negative chunk is designated uncorrectable if it is marked as an outlier by HDBSCAN or if c+=∅C_c^+= . Such chunks lack corrective targets and are only suppressed during training, as described in Section 3.3. Figure 3: Context-aware corrective matching in execution-context space. RedFlow represents each action chunk by its execution context (q~t,p¯t)( q_t,\; p_t), combining the robot’s proprioceptive state and task progress. Action chunks with similar contexts are grouped into execution-context clusters. Given a candidate failure chunk (red cross), RedFlow retrieves matched successful chunks from the corresponding cluster and uses them as corrective targets. The real-robot example illustrates the rationale: although the successful and failure chunks begin from similar contexts, only the matched successful chunk flattens the garment, while the failure chunk leaves it crumpled. 3.3 Adaptive Redirection Objective Applying the standard flow-matching objective (Eq. 3) uniformly clones both desirable and failure-prone behaviors. We instead introduce an asymmetric redirection objective. Theorem 1 (Bounded endpoint redirection). Consider a predicted clean action endpoint h∈h , a low-quality action chunk a−∈a^- , and a corrective target b∈b matched from a similar positive context. If b≠a−b≠ a^-, m>0m>0, and λsup≥λcor>0 _sup≥ _cor>0, define the endpoint energy ϕ(h)=λcor‖h−b‖2+λsup[m−‖h−a−‖2]+φ(h)= _cor\|h-b\|^2+ _sup[m-\|h-a^-\|^2]_+ (8) where [r]+=max(r,0)[r]_+= (r,0). The minimizer of ϕφ is the closest endpoint to the corrective target that remains outside the finite margin region around a−a^-. Equivalently, the solution is the projection of b onto the complement of the obstacle ball h:‖h−a−‖2<m\h:\|h-a^-\|^2<m\. The formal statement and proof are given in Appendix A, Theorem 6. As formalized in Appendix A, this objective induces a local Wasserstein push-pull transport: ℒattL_att anchors to positive endpoints, ℒsupL_sup bounds finite-range exclusion, and ℒcorL_cor directs corrective transport. Quality-weighted attraction. We first convert the estimated action-level score A^t A_t into a soft weight wt=σ(A^t/Tw)∈(0,1),w_t=σ( A_t/T_w)∈(0,1), (9) where Tw>0T_w>0 is a temperature. Confidently positive chunks receive weights close to one, while confidently negative chunks receive weights close to zero. We then use wtw_t to modulate the standard flow-matching loss: ℒatt=wt⋅‖vθ(xn,n,ot,l)−un‖2.L_att=w_t·\|v_θ(x_n,n,o_t,l)-u_n\|^2. (10) This term attracts the policy toward high-quality chunks while still providing a weak data-support signal for low-quality chunks. Failure suppression. For chunks estimated to be low-quality, attraction alone is insufficient: the policy should also reduce the tendency to reproduce failure-inducing actions. We therefore introduce a repulsive hinge loss on the reconstruction error et=‖x^0−at‖2,x^0=xn−n⋅vθ(xn,n,ot,l),e_t=\| x_0-a_t\|^2, x_0=x_n-n· v_θ(x_n,n,o_t,l), (11) where x^0 x_0 denotes the predicted clean action under the linear flow-matching interpolation. The suppression loss is ℒsup=λsup(1−wt)max(0,m−et),L_sup= _sup(1-w_t) (0,m-e_t), (12) where λsup>0 _sup>0 controls the suppression strength and m is an adaptive margin, implemented as a stop-gradient running average of the reconstruction error. Minimizing this hinge loss increases the distance between the predicted action and the negative chunk only when they are closer than m. Once the prediction is sufficiently far from the negative action, the repulsive penalty becomes inactive. Target-guided correction. Suppression prevents the policy from reproducing a negative chunk, but it does not specify where the local action distribution should move. For correctable failure chunks, we therefore add an attractive correction term toward the cluster-derived target at⋆a_t : ℒcor=ct⋅λcor(1−wt)‖x^0−at⋆‖2,L_cor=c_t· _cor(1-w_t)\| x_0-a_t \|^2, (13) where λcor>0 _cor>0 controls the correction strength and ct∈0,1c_t∈\0,1\ indicates whether chunk ata_t is a correctable failure chunk with an assigned target. This term should not be interpreted as cloning at⋆a_t as the exact corrective action for the state that produced ata_t. Rather, it provides a bounded endpoint-level bias that moves probability mass toward locally supported positive regions while the attraction term keeps training anchored to observed behavior data. Total objective. The final training objective is ℒ=(ot,at,l)∼,n[ℒatt+ℒsup+ℒcor].L=E_(o_t,a_t,l) ,\,n [L_att+L_sup+L_cor ]. (14) The three terms play complementary roles: quality-weighted attraction reinforces positive behavior, suppression discourages failure-inducing actions, and target-guided correction redirects correctable failures toward positive alternatives. RedFlow follows a single-iteration offline procedure summarized in Algorithm 1 of Appendix B. We first collect a fixed rollout buffer using the pretrained flow-matching VLA policy. We then derive chunk-level advantages and corrective targets using the procedure in Section 3.2. Finally, we optimize the policy on the frozen buffer using the adaptive redirection objective. 4 Experiment We conduct experiments to evaluate the effectiveness of RedFlow, an offline policy-improvement framework that redirects failure trajectories into action-level corrective supervision while also exploiting successful trajectories. Our experiments are designed to answer the following questions: (1) How does RedFlow compare to existing offline RL baselines on mixed-quality data? (2) How do the use of both successful and failed rollouts, Context-Aware Corrective Matching, and the Adaptive Redirection Objective each contribute to RedFlow? (3) How does RedFlow compare to online RL methods in terms of sample efficiency? (4) Does RedFlow transfer to real-robot tasks and yield consistent policy improvement? 4.1 Experimental Setup Simulation Experiments. We evaluate RedFlow on the four LIBERO suites [18]: Spatial, Object, Goal, and Long, each containing 10 tasks. We adopt π0 _0 [1] initialized from the πRL _RL [3] checkpoint as the base flow-matching VLA policy. Before offline RL fine-tuning, the base policy is trained on a pruned expert set expD_exp containing 58 demonstrations for Spatial/Object/Goal and 208 demonstrations for Long. Offline RL fine-tuning uses =exp∪rollD=D_exp _roll, where rollD_roll contains 1,536 mixed-quality rollout trajectories per suite. Progress estimates are produced by a pretrained General Reward Model (GRM) [29], smoothed over temporal chunks, and combined with proprioceptive states to implement Context-Aware Corrective Matching via HDBSCAN-based context clustering [21]. All methods share the same training protocol within each task suite. We report average success rates over 500 evaluation episodes per suite, and full hyperparameters are provided in Appendix C. Real-Robot Experiments. To evaluate RedFlow beyond simulation, we conduct real-world experiments on a dual-arm Agilex Cobot Magic robot equipped with three cameras, including one front-facing camera and two wrist-mounted cameras for visual input. We consider three tasks with different manipulation demands: clothes folding, object sweeping, and table cleaning (Fig. 4), covering dexterous bi-manual manipulation, tool-mediated interaction, and pick-and-place manipulation, respectively. We initialize from the official π0 _0 base checkpoint provided by OpenPi [1] and obtain the task-specific base policies by fine-tuning on expert demonstrations for 50,000 steps per task. The expert demonstration sets contain 600 demonstrations for clothes folding, 200 for object sweeping, and 100 for table cleaning. For offline RL training, we collect 200, 100, and 100 rollouts from the corresponding base policies for clothes folding, object sweeping, and table cleaning, respectively, yielding mixed buffers of successes and failures. We report average success rates over 100 evaluation episodes per task. Full task configurations and training details are provided in Appendix D. (a) Clothes Folding (b) Object Sweeping (c) Table Cleaning Figure 4: Real-robot experiment setups. We evaluate RedFlow on a dual-arm Agilex Cobot Magic robot across three tasks requiring different manipulation skills: (a) clothes folding for dexterous bi-manual manipulation, (b) object sweeping for tool-mediated interaction, and (c) table cleaning for pick-and-place manipulation. 4.2 Main Results Tab. 2 compares RedFlow with AWR [23] and DPO [25] on the four LIBERO suites. All methods share the same base policy, offline buffer D, and evaluation protocol. AWR imitates rollout actions reweighted by estimated advantage, while DPO learns from trajectory-level preferences without constructing action-level corrective targets. RedFlow achieves the best result on all four suites, lifting the average from 56.2% to 68.2% (+12.0 points), outperforming AWR by 5.9 points and DPO by 8.5 points. The largest gain appears on LIBERO-Goal (71.2% vs. 57.8% / 51.8%), showing the benefit of explicitly localizing failure-inducing chunks and redirecting them toward corrective targets retrieved from similar contexts. 4.3 Ablation Studies Tab. 2 ablates the three components of RedFlow in order: (i) rollout data composition, (i) Context-Aware Corrective Matching, and (i) the Adaptive Redirection Objective. Each variant removes one component while keeping the rest fixed. (i) Rollout data composition. The first block shows that successes and failures provide complementary signals. Removing failures drops the average to 68.0%, since the policy loses the negative evidence required for suppression and redirection; removing successes drops it further to 62.4%, since neither high-quality anchors nor corrective targets remain. Combining both is needed to reach 72.5%. (i) Context-Aware Corrective Matching. The second block tests whether corrective targets should be assigned indiscriminately. Without uncorrectable-failure separation, every failure chunk receives a target regardless of whether a similar successful context exists, causing the largest drop in the table (72.5% → 61.0%, with a 20.4-point loss on LIBERO-Goal). (i) Adaptive Redirection Objective. The third block isolates the two failure-side terms. Removing ℒsupL_sup alone (68.8%) leaves no mechanism to push probability mass away from failure-inducing actions; removing ℒcorL_cor alone (69.1%) removes the redirection signal toward retrieved corrective targets. Removing both reduces the objective to quality-weighted attraction and drops the average to 65.7%. Suppression and correction are therefore complementary rather than redundant: the former excludes failure-prone regions, the latter specifies where probability mass should move. Table 1: Success rates(%) on the LIBERO benchmark across four task suites. Method Spatial Object Goal Long Avg. Base Policy 63.6 61.6 48.6 50.8 56.2 AWR 71.2 66.8 57.8 53.4 62.3 DPO 65.8 69.8 51.8 51.2 59.7 RedFlow (Ours) 75.8 70.4 71.2 55.2 68.2 Table 2: Ablation studies on three LIBERO suites. Each number reports the average success rate(%). Method Spatial Object Goal Avg Rollout data composition w/o failure rollouts 71.4 67.4 65.2 68.0 w/o success rollouts 64.4 65.8 57.0 62.4 Context-Aware Corrective Matching w/o uncorrectable-failure separation 66.4 65.8 50.8 61.0 Adaptive Redirection Objective w/o ℒcorL_cor 70.4 68.4 68.6 69.1 w/o ℒsupL_sup 70.8 66.8 68.8 68.8 w/o ℒsupL_sup & ℒcorL_cor 63.8 62.8 70.4 65.7 !12 RedFlow (Ours) 75.8 70.4 71.2 72.5 Figure 5: Sample efficiency comparison on LIBERO-Spatial against on-policy RL baselines. Figure 6: Success rates on real-robot tasks. 4.4 Sample Efficiency Analysis We compare the sample efficiency of RedFlow against three on-policy RL baselines—PPO [26], GRPO [27], and DDPO [2]—on LIBERO-Spatial. As shown in Fig. 6, RedFlow reaches 75.8% success (red dashed line) using only 1,536 offline trajectories and no additional environment interaction. The on-policy baselines collect 1,024 fresh rollouts at every update step; reaching the same success level requires roughly 13 / 16 / 24 update steps for PPO / GRPO / DDPO, corresponding to approximately 13K / 16K / 24K rollout trajectories—an order of magnitude more than RedFlow. This shows that structured failure reuse can substitute for a substantial portion of on-policy interaction. 4.5 Real-Robot Results Fig. 6 reports success rates on the three real-robot tasks. RedFlow achieves the highest average success rate of 74.7%, outperforming all baselines on every task. The largest gain appears on clothes folding, where RedFlow improves the base policy from 36.0% to 67.0%, a 31.0-point absolute improvement that confirms the framework transfers to challenging bimanual manipulation. Qualitative Recovery Analysis. Fig. 7 illustrates how RedFlow learns recovery behavior on clothes folding. The base policy fails when the T-shirt falls out of the right arm’s reach: the right arm keeps attempting to grasp it but cannot recover. RedFlow instead executes a corrective retry—using the left arm to pull the T-shirt back into a reachable configuration before resuming the fold. This behavior emerges purely from offline buffer reuse: Context-Aware Corrective Matching retrieves left-arm pulling actions from similar progress–state contexts as corrective targets, and the Adaptive Redirection Objective redirects the failed grasp attempts toward them. Figure 7: Qualitative correction behavior on clothes folding. The base policy fails when the T-shirt falls out of the right arm’s reach, while RedFlow recovers by using the left arm to pull the cloth back and continue folding. 5 Conclusion We presented RedFlow, an offline post-training framework that converts deployment failures of flow-matching VLA policies into action-level corrective supervision. Context-Aware Corrective Matching identifies recoverable failure-inducing chunks and retrieves successful alternatives from similar execution contexts, while the Adaptive Redirection Objective integrates these signals by reinforcing high-quality actions, suppressing undesirable ones, and redirecting recoverable failures toward retrieved targets. On LIBERO and three real-robot tasks, RedFlow surpasses offline baselines by clear margins and matches strong on-policy methods with roughly an order of magnitude fewer trajectories, lifting real-world success from 56.7% to 74.7%. These results establish structured failure reuse as a sample-efficient direction for post-training generalist VLA policies. Future work will investigate uncertainty-aware target assignment for out-of-distribution states, longer-horizon tasks, and extensions to iterative offline–online improvement. References [1] K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al. (2024) π0 _0: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164. Cited by: Appendix C, Appendix D, §1, §2, §4.1, §4.1. [2] K. Black, M. Janner, Y. Du, I. Kostrikov, and S. Levine (2024) Training diffusion models with reinforcement learning. External Links: 2305.13301, Link Cited by: §4.4. [3] K. Chen, Z. Liu, T. Zhang, Z. Guo, S. Xu, H. Lin, H. Zang, X. Li, Q. Zhang, Z. Yu, G. Fan, T. Huang, Y. Wang, and C. Yu (2026) πRL _ RL: Online rl fine-tuning for flow-based vision-language-action models. External Links: 2510.25889, Link Cited by: Appendix C, §4.1. [4] Y. Chen, Y. Jian, X. Dong, S. Cao, J. Wu, P. Vela, B. E. Lundell, and D. Chen (2026) VISTA: enhancing visual conditioning via track-following preference optimization in vision-language-action models. External Links: 2602.05049, Link Cited by: §1, §2. [5] K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela (2024) KTO: model alignment as prospect theoretic optimization. External Links: 2402.01306, Link Cited by: §2. [6] K. Frans, S. Park, P. Abbeel, and S. Levine (2025) Diffusion guidance is a controllable policy improvement operator. arXiv preprint arXiv:2505.23458. Cited by: §2. [7] D. Huang, Z. Fang, T. Zhang, Y. Li, L. Zhao, and C. Xia (2025) Co-rft: efficient fine-tuning of vision-language-action models through chunked offline reinforcement learning. arXiv preprint arXiv:2508.02219. Cited by: §2. [8] P. Intelligence, A. Amin, R. Aniceto, A. Balakrishna, K. Black, K. Conley, G. Connors, J. Darpinian, K. Dhabalia, J. DiCarlo, D. Driess, M. Equi, A. Esmail, Y. Fang, C. Finn, C. Glossop, T. Godden, I. Goryachev, L. Groom, H. Hancock, K. Hausman, G. Hussein, B. Ichter, S. Jakubczak, R. Jen, T. Jones, B. Katz, L. Ke, C. Kuchi, M. Lamb, D. LeBlanc, S. Levine, A. Li-Bell, Y. Lu, V. Mano, M. Mothukuri, S. Nair, K. Pertsch, A. Z. Ren, C. Sharma, L. X. Shi, L. Smith, J. T. Springenberg, K. Stachowicz, W. Stoeckle, A. Swerdlow, J. Tanner, M. Torne, Q. Vuong, A. Walling, H. Wang, B. Williams, S. Yoo, L. Yu, U. Zhilinsky, and Z. Zhou (2025) π0.6∗π^*_0.6: A vla that learns from experience. External Links: 2511.14759, Link Cited by: §1, §2. [9] P. Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, M. Y. Galliker, D. Ghosh, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, D. LeBlanc, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, A. Z. Ren, L. X. Shi, L. Smith, J. T. Springenberg, K. Stachowicz, J. Tanner, Q. Vuong, H. Walke, A. Walling, H. Wang, L. Yu, and U. Zhilinsky (2025) π0.5 _0.5: A vision-language-action model with open-world generalization. External Links: 2504.16054, Link Cited by: §1, §2. [10] M. Kelly, C. Sidrane, K. Driggs-Campbell, and M. J. Kochenderfer (2019) HG-dagger: interactive imitation learning with human experts. In 2019 International Conference on Robotics and Automation (ICRA), p. 8077–8083. Cited by: §1, §2. [11] M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. P. Foster, P. R. Sanketi, Q. Vuong, et al. (2025) OpenVLA: an open-source vision-language-action model. In Conference on Robot Learning, p. 2679–2713. Cited by: §2. [12] T. Lee, A. Wagenmaker, K. Pertsch, P. Liang, S. Levine, and C. Finn (2026) RoboReward: general-purpose vision-language reward models for robotics. External Links: 2601.00675, Link Cited by: §2. [13] K. Lei, H. Li, D. Yu, Z. Wei, L. Guo, Z. Jiang, Z. Wang, S. Liang, and H. Xu (2025) Rl-100: performant robotic manipulation with real-world reinforcement learning. arXiv preprint arXiv:2510.14830. Cited by: §2. [14] S. Levine (2018) Reinforcement learning and control as probabilistic inference: tutorial and review. External Links: 1805.00909, Link Cited by: §1. [15] H. Li, Y. Zuo, J. Yu, Y. Zhang, Z. Yang, K. Zhang, X. Zhu, Y. Zhang, T. Chen, G. Cui, D. Wang, D. Luo, Y. Fan, Y. Sun, J. Zeng, J. Pang, S. Zhang, Y. Wang, Y. Mu, B. Zhou, and N. Ding (2025) SimpleVLA-rl: scaling vla training via reinforcement learning. External Links: 2509.09674, Link Cited by: §1, §2. [16] A. Liang, Y. Korkmaz, J. Zhang, M. Hwang, A. Anwar, S. Kaushik, A. Shah, A. S. Huang, L. Zettlemoyer, D. Fox, Y. Xiang, A. Li, A. Bobu, A. Gupta, S. Tu, E. Biyik, and J. Zhang (2026) Robometer: scaling general-purpose robotic reward models via trajectory comparisons. External Links: 2603.02115, Link Cited by: §2. [17] Y. Lipman, R. T. Chen, H. Ben-Hamu, M. Nickel, and M. Le (2023) Flow matching for generative modeling. In 11th International Conference on Learning Representations, ICLR 2023, Cited by: §3.1. [18] B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone (2023) LIBERO: benchmarking knowledge transfer for lifelong robot learning. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36, p. 44776–44791. External Links: Link Cited by: Appendix C, §4.1. [19] G. Lu, W. Guo, C. Zhang, Y. Zhou, H. Jiang, Z. Gao, Y. Tang, and Z. Wang (2025) VLA-rl: towards masterful and general robotic manipulation with scalable reinforcement learning. External Links: 2505.18719, Link Cited by: §1, §2. [20] J. Luo, C. Xu, J. Wu, and S. Levine (2025) Precise and dexterous robotic manipulation via human-in-the-loop reinforcement learning. Science Robotics 10 (105), p. eads5033. Cited by: §1. [21] L. McInnes and J. Healy (2017) Accelerated hierarchical density based clustering. In 2017 IEEE International Conference on Data Mining Workshops (ICDMW), Vol. , p. 33–42. External Links: Document Cited by: Appendix B, §3.2, §4.1. [22] A. O’Neill, A. Rehman, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jain, et al. (2024) Open x-embodiment: robotic learning datasets and rt-x models: open x-embodiment collaboration 0. In 2024 IEEE International Conference on Robotics and Automation (ICRA), p. 6892–6903. Cited by: §2. [23] X. B. Peng, A. Kumar, G. Zhang, and S. Levine (2019) Advantage-weighted regression: simple and scalable off-policy reinforcement learning. External Links: 1910.00177, Link Cited by: §1, §2, §2, §4.2. [24] J. Peters and S. Schaal (2007) Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th international conference on Machine learning, p. 745–750. Cited by: §1. [25] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn (2023) Direct preference optimization: your language model is secretly a reward model. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36, p. 53728–53741. External Links: Link Cited by: §4.2. [26] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. External Links: 1707.06347, Link Cited by: §4.4. [27] Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo (2024) DeepSeekMath: pushing the limits of mathematical reasoning in open language models. External Links: 2402.03300, Link Cited by: §4.4. [28] Q. Shou, F. Zhu, S. Chen, P. Yan, Z. Yan, Y. Miao, X. Pang, Z. Hong, R. Shi, H. Huang, J. Zhang, and S. Guo (2026) HALO: a unified vision-language-action model for embodied multimodal chain-of-thought reasoning. External Links: 2602.21157, Link Cited by: §2. [29] H. Tan, S. Chen, Y. Xu, Z. Wang, Y. Ji, C. Chi, Y. Lyu, Z. Zhao, X. Chen, P. Co, S. Xie, G. Yao, P. Wang, Z. Wang, and S. Zhang (2025) Robo-dopamine: general process reward modeling for high-precision robotic manipulation. External Links: 2512.23703, Link Cited by: Appendix B, §2, §3.2, §4.1. [30] O. M. Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, T. Kreiman, C. Xu, J. Luo, Y. L. Tan, L. Y. Chen, P. Sanketi, Q. Vuong, T. Xiao, D. Sadigh, C. Finn, and S. Levine (2024) Octo: an open-source generalist robot policy. External Links: 2405.12213, Link Cited by: §2. [31] W. Xia, Y. Yang, H. Wu, X. Ma, T. Kong, and D. Hu (2025) Human-assisted robotic policy refinement via action preference optimization. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, Cited by: §1, §2. [32] C. Yu, Y. Wang, Z. Guo, H. Lin, S. Xu, H. Zang, Q. Zhang, Y. Wu, C. Zhu, J. Hu, et al. (2025) Rlinf: flexible and efficient large-scale reinforcement learning via macro-to-micro flow transformation. arXiv preprint arXiv:2509.15965. Cited by: §2. [33] T. Zhang, C. Yu, S. Su, and Y. Wang (2025) ReinFlow: fine-tuning flow matching policy with online reinforcement learning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, Cited by: §1, §2. [34] Z. Zhang, K. Zheng, Z. Chen, J. Jang, Y. Li, S. Han, C. Wang, M. Ding, D. Fox, and H. Yao (2024) Grape: generalizing robot policy via preference alignment. arXiv preprint arXiv:2411.19309. Cited by: §1, §2. [35] F. Zhu, Z. Yan, Z. Hong, Q. Shou, X. Ma, and S. Guo (2025) Wmpo: world model-based policy optimization for vision-language-action models. arXiv preprint arXiv:2511.09515. Cited by: §2. [36] B. Zitkovich, T. Yu, S. Xu, P. Xu, T. Xiao, F. Xia, J. Wu, P. Wohlhart, S. Welker, A. Wahid, Q. Vuong, V. Vanhoucke, H. Tran, R. Soricut, A. Singh, J. Singh, P. Sermanet, P. R. Sanketi, G. Salazar, M. S. Ryoo, K. Reymann, K. Rao, K. Pertsch, I. Mordatch, H. Michalewski, Y. Lu, S. Levine, L. Lee, T. E. Lee, I. Leal, Y. Kuang, D. Kalashnikov, R. Julian, N. J. Joshi, A. Irpan, B. Ichter, J. Hsu, A. Herzog, K. Hausman, K. Gopalakrishnan, C. Fu, P. Florence, C. Finn, K. A. Dubey, D. Driess, T. Ding, K. M. Choromanski, X. Chen, Y. Chebotar, J. Carbajal, N. Brown, A. Brohan, M. G. Arenas, and K. Han (2023-06–09 Nov) RT-2: vision-language-action models transfer web knowledge to robotic control. In Proceedings of The 7th Conference on Robot Learning, J. Tan, M. Toussaint, and K. Darvish (Eds.), Proceedings of Machine Learning Research, Vol. 229, p. 2165–2183. External Links: Link Cited by: §1, §2. Appendix Appendix A Bounded Corrective Redirection as Constrained Wasserstein Policy Transport This appendix formalizes the endpoint-redirection principle stated in Theorem 1 and gives a broader variational interpretation of RedFlow from the perspective of conditional action-distribution transport. Rather than treating positive attraction, negative suppression, and corrective redirection as independent heuristics, we show that they arise as particle-level realizations of a local transport principle whose first variation yields a signed push–pull field. Throughout, =ℝK×DA=R^K× D denotes the action-chunk space and ∥⋅∥\|·\| denotes the Frobenius norm. For any scalar r, we write [r]+=maxr,0[r]_+= \r,0\. Scope. The theory characterizes the geometry of the RedFlow update conditional on the estimated chunk scores and assigned corrective targets. It establishes that the objective implements bounded exclusion from low-quality endpoints and constructive transport toward locally supported positive references. Standing assumptions. We assume =ℝK×DA=R^K× D with squared Euclidean cost, local behavior measures have finite second moments, soft gates are measurable and bounded in (0,1)(0,1), and corrective targets are assigned only to clusters with non-empty estimated-positive support. Hinge derivatives are understood almost everywhere; the boundary ‖a−a−‖2=m\|a-a^-\|^2=m has zero measure under any absolutely continuous policy measure. A.1 Conditional Action Measures and Soft Gating For a local context z induced by the observation, instruction, and progress–state feature, let μθz∈2() _θ^z _2(A) denote the conditional action distribution induced by the flow policy. Let νβz _β^z denote the local behavior distribution represented by the offline buffer. For the population-level transport energy, we use a smoothed empirical behavior reference νβ,ϵz _β,ε^z, which provides a regular density-level proxy for the local buffer distribution. In finite samples, νβ,ϵz _β,ε^z is evaluated through the same local buffer points used by the practical objective. The estimated chunk-level advantage induces a soft gate w(a,z)=σ(A^(a,z)/Tw),w(a,z)=σ ( A(a,z)/T_w ), (15) which decomposes the local behavior reference into soft positive and negative components: dν+,wz(a)=w(a,z)Z+zdνβ,ϵz(a),dν−,wz(a)=1−w(a,z)Z−zdνβ,ϵz(a),d _+,w^z(a)= w(a,z)Z_+^z\,d _β,ε^z(a), _-,w^z(a)= 1-w(a,z)Z_-^z\,d _β,ε^z(a), (16) where Z+z=∫w(a,z)dνβ,ϵz(a),Z−z=∫(1−w(a,z))dνβ,ϵz(a).Z_+^z= _Aw(a,z)\,d _β,ε^z(a), Z_-^z= _A (1-w(a,z) )\,d _β,ε^z(a). (17) The sample-level weights wt=σ(A^t/Tw)w_t=σ( A_t/T_w) and 1−wt1-w_t used in RedFlow are Monte Carlo realizations of this soft measure decomposition. A low-quality chunk is correctable only if it is assigned to a valid progress–state cluster whose estimated-positive subset is non-empty. For such a cluster cC_c, define c+=i∈c:A^i>0.C_c^+=\i _c: A_i>0\. (18) For uncorrectable low-quality chunks, no corrective target is defined and only bounded suppression is applied. A.2 Corrective Targets as Advantage-Tilted Fréchet Means Theorem 2 (Corrective target as empirical Fréchet mean). For a cluster cC_c with non-empty positive subset c+C_c^+, define bz=∑i∈c+αiai,αi=exp(A^i/κ)∑j∈c+exp(A^j/κ).b_z= _i _c^+ _ia_i, _i= ( A_i/κ) _j _c^+ ( A_j/κ). (19) Then bzb_z is the unique minimizer of the empirical weighted Fréchet problem minb∈∑i∈c+αi‖b−ai‖2. _b _i _c^+ _i\|b-a_i\|^2. (20) It is therefore the empirical squared-cost barycenter of the advantage-tilted positive support in the local progress–state cluster. Proof. The weights satisfy αi>0 _i>0 and ∑iαi=1 _i _i=1. The objective in Eq. (20) is strictly convex in the Euclidean action space. Its stationary condition is ∑i∈c+αi(bz−ai)=0,bz=∑i∈c+αiai, _i _c^+ _i(b_z-a_i)=0, b_z= _i _c^+ _ia_i, (21) which is the unique global minimizer. ∎ This empirical target is a plug-in estimator of an advantage-tilted local positive mean. Let the oracle advantage-tilted positive reference measure be ν+⋆(a∣z)∝νβ,+(a∣z)exp(A⋆(z,a)/κ), _+ (a z) _β,+(a z) (A (z,a)/κ), (22) where νβ,+ _β,+ is the positive component of the local behavior distribution and A⋆A is the latent local advantage. The squared-cost Fréchet mean of this tilted measure is bz⋆=argminb∈∫‖b−a‖2dν+⋆(a∣z).b_z = _b _A\|b-a\|^2\,d _+ (a z). (23) Under the Euclidean action metric, bz⋆=a∼ν+⋆(⋅∣z)[a].b_z =E_a _+ (· z)[a]. (24) In Euclidean action spaces, this expectation is exact for squared cost; on non-Euclidean action manifolds, the same construction generalizes to the corresponding manifold Fréchet mean. This explains why corrective redirection pulls toward a weighted positive barycenter rather than toward an arbitrary successful chunk. A.3 A Local Transport Energy We interpret local policy improvement as a proximal transport step over conditional action distributions. Building upon the Jordan–Kinderlehrer–Otto scheme, consider the one-step variational update μk+1z=argminμ∈2()12τW22(μ,μkz)+ℰz(μ),μ^z_k+1= _μ _2(A) 12τW_2^2(μ,μ^z_k)+E_z(μ), (25) where W2W_2 is the quadratic Wasserstein distance and ℰzE_z is the local transport energy ℰz(μ)= _z(μ)= czλcor2∫‖a−bz‖2dμ(a) \; c_z _cor2 _A\|a-b_z\|^2\,dμ(a) +λsup∫[m−‖a−a−‖2]+dν−,wz(a−)dμ(a) + _sup _A _A [m-\|a-a^-\|^2 ]_+\,dν^z_-,w(a^-)\,dμ(a) +λbc2∫w(a−,z)‖a−a−‖2dνβ,ϵz(a−)dμ(a). + _bc2 _A _Aw(a^-,z)\|a-a^-\|^2\,dν^z_β,ε(a^-)\,dμ(a). (26) Here cz∈0,1c_z∈\0,1\ indicates whether a corrective target exists. The first term attracts the transported action distribution toward the advantage-tilted positive reference bzb_z. The second term treats low-quality chunks as finite-range obstacles rather than global repulsive charges. The third term is an empirical endpoint-anchoring energy: it keeps transported particles tied to high-quality behavior endpoints through a weighted quadratic transport cost. This particle-based anchoring form matches the squared endpoint regression structure used by flow-matching training. As τ→0τ→ 0, the proximal sequence formally induces a Wasserstein gradient flow ∂sμsz+∇a⋅(μszusz)=0,usz(a)=−∇aδℰzδμ(a), _sμ^z_s+ _a·(μ^z_su^z_s)=0, u^z_s(a)=- _a _zδμ(a), (27) where s denotes the policy-transport time, distinct from the flow-matching timestep n. A.4 Push–Pull Velocity from First Variation The local transport energy in Eq. (A.3) induces a transport velocity field over action space. Its first variation gives the desired push–pull structure. Theorem 3 (Push–pull velocity field). For almost every a∈a , the Wasserstein steepest-descent velocity induced by ℰzE_z is uz(a)= u^z(a)= czλcor(bz−a) \;c_z _cor(b_z-a) +2λsup∫‖a−a−‖2<m(a−a−)dν−,wz(a−) +2 _sup _A1\\|a-a^-\|^2<m\(a-a^-)\,dν^z_-,w(a^-) +λbc∫w(a−,z)(a−a)dνβ,ϵz(a−). + _bc _Aw(a^-,z)(a^--a)\,dν^z_β,ε(a^-). (28) Proof. The first variation of the corrective attraction term is czλcor2‖a−bz‖2, c_z _cor2\|a-b_z\|^2, (29) whose negative spatial gradient is czλcor(bz−a)c_z _cor(b_z-a). The first variation of the obstacle term is λsup∫[m−‖a−a−‖2]+dν−,wz(a−). _sup _A [m-\|a-a^-\|^2 ]_+\,dν^z_-,w(a^-). (30) For almost every a, its negative spatial gradient is 2λsup∫‖a−a−‖2<m(a−a−)dν−,wz(a−).2 _sup _A1\\|a-a^-\|^2<m\(a-a^-)\,dν^z_-,w(a^-). (31) The first variation of the empirical endpoint-anchoring term is λbc2∫w(a−,z)‖a−a−‖2dνβ,ϵz(a−), _bc2 _Aw(a^-,z)\|a-a^-\|^2\,dν^z_β,ε(a^-), (32) whose negative spatial gradient is λbc∫w(a−,z)(a−a)dνβ,ϵz(a−). _bc _Aw(a^-,z)(a^--a)\,dν^z_β,ε(a^-). (33) Combining the three components gives the stated velocity field. ∎ Theorem 3 gives the structural justification for RedFlow. Positive and negative samples do not enter symmetrically. Positive samples define constructive transport destinations through bzb_z, whereas negative samples define finite-range obstacles that exclude locally undesirable action regions. Therefore, the update is not pure repulsion from failures; it redirects probability mass from failure neighborhoods toward locally supported positive references while remaining anchored to the behavior support. The margin m has a precise geometric meaning in this view. It specifies the interaction radius of the obstacle ball ℬ(a−,m)=a:‖a−a−‖2<m.B(a^-, m)=\a:\|a-a^-\|^2<m\. (34) Inside this ball, the negative chunk induces a repulsive transport component. Outside it, the negative chunk exerts no force. This finite-range structure is essential in continuous action spaces: negative evidence identifies where probability mass should not remain, but does not specify a global direction in which the policy should move. A.5 Flow-Matching Endpoint Realization Theorem 4 (Endpoint attraction as velocity regression). Under the linear endpoint parameterization x^0=xn−nvθ(xn,n,z) x_0=x_n-nv_θ(x_n,n,z), endpoint attraction toward any y∈y is equivalent, up to the positive factor n2n^2, to flow-matching velocity regression toward the linear-path velocity uny(xn,z)=(xn−y)/nu_n^y(x_n,z)=(x_n-y)/n. Proof. For an endpoint target y∈y , define the corresponding linear-path target velocity as uny(xn,z)=xn−yn.u_n^y(x_n,z)= x_n-yn. (35) Under the reverse-time endpoint parameterization used by the policy, the denoised prediction is x^0=xn−nvθ(xn,n,z). x_0=x_n-nv_θ(x_n,n,z). (36) Therefore, ‖x^0−y‖2=n2‖vθ(xn,n,z)−uny(xn,z)‖2.\| x_0-y\|^2=n^2 \|v_θ(x_n,n,z)-u_n^y(x_n,z) \|^2. (37) Note that while the minimizers are identical, the scale factor n2n^2 implies that regressing in the velocity space implicitly schedules the strength of endpoint attraction. This time-dependent scaling is a standard feature of flow-matching objectives, naturally down-weighting the effective endpoint constraint at early flow stages (n≈1n≈ 1, closer to noise) and strengthening it near the generation boundary (n≈0n≈ 0). ∎ Thus, endpoint-space attraction toward y can be implemented as target-velocity regression at the same noisy anchor. Choosing y=aty=a_t gives empirical behavior anchoring through standard flow-matching regression to observed buffer endpoints. Choosing y=bzty=b_z_t gives corrective attraction toward the local positive reference. The hinge term acts as an endpoint-space obstacle potential around negative actions. At the endpoint level, the suppression potential for a negative chunk ata_t is ϕsup(x^0,at)=[m−‖x^0−at‖2]+. _sup( x_0,a_t)=[m-\| x_0-a_t\|^2]_+. (38) For ‖x^0−at‖2≠m\| x_0-a_t\|^2≠ m, its negative endpoint gradient is −∇x^0ϕsup=2‖x^0−at‖2<m(x^0−at).- _ x_0 _sup=21\\| x_0-a_t\|^2<m\( x_0-a_t). (39) Thus, the endpoint-level suppression direction pushes the prediction away from ata_t only inside the margin and vanishes outside the obstacle ball. Combining the transport components at the particle level yields the RedFlow endpoint surrogate ℒt _t =wt‖vθ(xn,n,zt)−unat(xn,zt)‖2 =w_t\|v_θ(x_n,n,z_t)-u_n^a_t(x_n,z_t)\|^2 (40) +λsup(1−wt)[m−‖x^0−at‖2]+ + _sup(1-w_t)[m-\| x_0-a_t\|^2]_+ +ctλcor(1−wt)‖x^0−bzt‖2. +c_t _cor(1-w_t)\| x_0-b_z_t\|^2. Here ct∈0,1c_t∈\0,1\ indicates whether the negative chunk has an assigned corrective target. When wtw_t is large, the sample primarily contributes to empirical behavior anchoring; when wtw_t is small, it primarily contributes to finite-range suppression and, if ct=1c_t=1, corrective redirection. The first term is the flow-matching realization of the empirical endpoint-anchoring energy. For a sampled behavior endpoint ata_t, the endpoint quadratic cost wt‖x^0−at‖2w_t\| x_0-a_t\|^2 induces the attraction direction wt(at−x^0)w_t(a_t- x_0). Under the linear endpoint parameterization x^0=xn−nvθ(xn,n,zt) x_0=x_n-nv_θ(x_n,n,z_t) and unat(xn,zt)=(xn−at)/nu_n^a_t(x_n,z_t)=(x_n-a_t)/n, we have ‖x^0−at‖2=n2‖vθ(xn,n,zt)−unat(xn,zt)‖2.\| x_0-a_t\|^2=n^2\|v_θ(x_n,n,z_t)-u_n^a_t(x_n,z_t)\|^2. (41) Thus, weighted velocity regression is the flow-matching form of weighted endpoint attraction. The second term realizes finite-range endpoint exclusion around low-quality chunks, while the third term realizes corrective attraction toward the advantage-tilted positive reference. Together, Eq. (40) instantiates the push–pull transport geometry in Theorem 3 at the particle level. A.6 Energy Dissipation and Local Stationary Convergence While establishing global convergence for non-linear deep neural networks over continuous action spaces is generally intractable, the Wasserstein gradient flow formulation provides a strong geometric guarantee of local stationary convergence for the RedFlow update. Theorem 5 (Energy Dissipation and Stationary Convergence). Let μsz _s^z be the solution to the idealized Wasserstein gradient flow ∂sμsz+∇a⋅(μszusz)=0 _s _s^z+ _a·( _s^zu_s^z)=0 driven by the local transport energy ℰz(μ)E_z(μ) in Eq. (A.3). Assume the initial energy is finite. Then the energy is strictly non-increasing along the flow: dsℰz(μsz)=−∫‖usz(a)‖2dμsz(a)≤0. ddsE_z(μ^z_s)=- _A\|u^z_s(a)\|^2\,dμ^z_s(a)≤ 0. (42) Consequently, under this non-parametric particle dynamics, the distribution converges to a stationary configuration where the push-pull velocity field vanishes almost everywhere. Proof. By the chain rule for Wasserstein gradient flows, the time derivative of the energy functional along its own gradient flow is given by the negative squared L2(μsz)L^2( _s^z) norm of the minimal-norm subdifferential. Since usz(a)=−∇aδℰzδμ(a)u^z_s(a)=- _a _zδμ(a), we have: dsℰz(μsz)=∫δℰzδμ(a)∂sμsz(a)da=−∫‖∇aδℰzδμ(a)‖2dμsz(a)≤0. ddsE_z(μ^z_s)= _A _zδμ(a) _sμ^z_s(a)\,da=- _A \| _a _zδμ(a) \|^2dμ^z_s(a)≤ 0. (43) The local transport energy ℰz(μ)E_z(μ) is bounded from below by 0, as it is a pure sum of non-negative quadratic distances and non-negative hinge losses (Eq. (A.3)). Because ℰzE_z is bounded from below and monotonically decreasing, the limit lims→∞ℰz(μsz) _s→∞E_z(μ^z_s) exists and is finite. Consequently, the dissipation rate must decay to zero: lims→∞∫‖usz(a)‖2dμsz(a)=0. _s→∞ _A\|u^z_s(a)\|^2\,dμ^z_s(a)=0. (44) This implies that the probability mass strictly stabilizes, converging to a stationary distribution μ∞zμ^z_∞ where the constructive attraction and finite-range suppression forces perfectly balance. ∎ This theorem provides the geometric safeguard for RedFlow. While practical deep neural network optimization involves complexities like finite capacity and mini-batch stochasticity that preclude absolute convergence guarantees, this continuous-time particle view ensures that the underlying objective is intrinsically dissipative. It demonstrates that despite the non-convexity introduced by the finite-range obstacle term (ℒsupL_sup), the fundamental transport forces are designed to optimally balance behavioral anchoring, physical safety constraints, and constructive redirection without inherent oscillatory dynamics. A.7 Proof of the Endpoint Redirection Principle We now prove the single-obstacle endpoint result used in Theorem 1. This result characterizes the interaction between bounded suppression and target-guided correction before considering the population-level transport view. Theorem 6 (Single-obstacle endpoint geometry). Consider a single negative obstacle a−∈a^- , a corrective target bz∈b_z , and an endpoint variable h∈h . Assume bz≠a−b_z≠ a^-, m>0m>0, and λsup≥λcor>0 _sup≥ _cor>0. The minimizer of ϕ(h)=λcor‖h−bz‖2+λsup[m−‖h−a−‖2]+φ(h)= _cor\|h-b_z\|^2+ _sup[m-\|h-a^-\|^2]_+ (45) coincides with the projection of bzb_z onto the complement of the obstacle ball: h⋆=bz,‖bz−a−‖2≥m,a−+mbz−a−‖bz−a−‖,‖bz−a−‖2<m.h = casesb_z,&\|b_z-a^-\|^2≥ m,\\[4.0pt] a^-+ m\, b_z-a^-\|b_z-a^-\|,&\|b_z-a^-\|^2<m. cases (46) Equivalently, h⋆h solves the constrained projection problem minh∈∥h−bz∥2s.t.∥h−a−∥2≥m. _h \|h-b_z\|^2 .t. \|h-a^-\|^2≥ m. (47) Proof. If ‖bz−a−‖2≥m\|b_z-a^-\|^2≥ m, the hinge term is inactive at h=bzh=b_z, and h=bzh=b_z minimizes both terms in Eq. (45). Now suppose d=‖bz−a−‖<md=\|b_z-a^-\|< m. For fixed r=‖h−a−‖r=\|h-a^-\|, the hinge term depends only on r, and the attraction term is minimized on the ray h=a−+rbz−a−d.h=a^-+r b_z-a^-d. (48) Indeed, among points at radius r from a−a^-, this ray point maximizes the inner product with bz−a−b_z-a^- by Cauchy–Schwarz and therefore minimizes ‖h−bz‖2\|h-b_z\|^2; no off-ray point can achieve a smaller objective. Inside the obstacle ball, the problem therefore reduces to the one-dimensional objective ψ(r)=λcor(d−r)2+λsup(m−r2),0≤r<m.ψ(r)= _cor(d-r)^2+ _sup(m-r^2), 0≤ r< m. (49) Its derivative is ψ′(r)=2(λcor−λsup)r−2λcord.ψ (r)=2( _cor- _sup)r-2 _cord. (50) Since λsup≥λcor>0 _sup≥ _cor>0 and d>0d>0, we have ψ′(r)<0ψ (r)<0 for all 0≤r<m0≤ r< m. Hence the objective decreases up to the boundary r=mr= m. Outside the obstacle ball, the hinge term vanishes, and the closest feasible point to bzb_z is precisely the boundary projection in Eq. (46). ∎ Consequently, hinge suppression implements bounded exclusion, while the correction term selects the closest safe endpoint aligned with the local positive reference. The endpoint subsystem therefore does not induce arbitrary repulsion from failures; it redirects the prediction toward the nearest margin-satisfying point consistent with the corrective target. This proves the endpoint principle summarized in Theorem 1. Theorem 7 (Bounded redirection of the corrective subsystem). For a correctable negative chunk with ct=1c_t=1 and fixed weight wt<1w_t<1, consider the corrective subsystem of the sample objective, isolated by analyzing only the terms acting away from the failure anchor: λsup(1−wt)[m−‖x^0−at‖2]++λcor(1−wt)‖x^0−bzt‖2 _sup(1-w_t)[m-\| x_0-a_t\|^2]_++ _cor(1-w_t)\| x_0-b_z_t\|^2 This subsystem shares the exact same minimizers as Theorem 6 with a−=ata^-=a_t and bz=bztb_z=b_z_t. Hence, the repulsive force vanishes outside the margin ball around ata_t, and inside the margin, the correction term selects the closest margin-satisfying endpoint aligned with the local positive reference. The final prediction naturally interpolates between this bounded corrective target and the empirical anchor ata_t governed by the omitted attraction term ℒattL_att. Proof. The positive scalar (1−wt)(1-w_t) does not change minimizers. The remaining objective is exactly Eq. (45) after substituting a−=ata^-=a_t and bz=bztb_z=b_z_t. ∎ A.8 Interpretation The analysis characterizes RedFlow as a local transport procedure in continuous action space. Positive chunks provide constructive destinations, negative chunks provide finite-range exclusion regions, and behavior anchoring keeps the transported distribution tied to the local buffer support. Flow matching provides the velocity-parameterized mechanism for realizing the resulting endpoint transport. In multimodal neighborhoods, the same framework can be extended by replacing the single barycentric target with mixture-valued positive references. Appendix B Training Details Overall training pipeline. RedFlow follows a single-iteration offline post-training pipeline for flow-based VLA policies. We first obtain a task-specific base policy by fine-tuning π0 _0 on expert demonstrations. To fine-tune π0 _0 and obtain the base policy, we use 8×8×H20 GPUs. We then collect a fixed offline rollout buffer with the base policy, including both successful and failed trajectories. Given the frozen buffer, we estimate chunk-level progress with a pretrained GRM, assign action-level quality labels, construct corrective targets with progress–state-aware HDBSCAN clustering, and finally update the policy using the adaptive redirection objective. Progress estimation and corrective target assignment. For both simulation and real-robot experiments, we use the pretrained GRM [29] to estimate task progress and apply the progress–state-aware corrective targeting procedure described in Section 3.2. Progress estimates are smoothed over temporal chunks and combined with trajectory outcomes to obtain the proxy advantage A^t A_t. We then cluster action chunks with HDBSCAN [21] in the normalized progress–state space and assign corrective targets from co-clustered positive chunks when available. Detailed HDBSCAN and progress-estimation hyperparameters are provided separately for simulation and real-robot experiments in Tables 4 and 6. Offline policy optimization. The policy is optimized with the adaptive redirection objective in Eq. (14), which combines quality-weighted attraction, failure suppression, and target-guided correction. The loss coefficients and optimization hyperparameters are reported in the corresponding simulation and real-robot implementation tables. Algorithm. The full training procedure is summarized in Algorithm 1. Algorithm 1 RedFlow training pipeline 1:Pretrained policy πθ _θ, optional demonstrations expD_exp, GRM R, task instruction l, epochs E 2:// Phase 1: Data collection 3:roll←τii=1ND_roll←\ _i\_i=1^N via rollouts of πθ _θ 4:←exp∪rollD _exp _roll 5:// Phase 2: Progress-state Clustering for corrective targets 6:for at∈a_t do 7: p¯t←12W+1∑j=t−Wt+WR(oj,l) p_t← 12W+1 _j=t-W^t+WR(o_j,l) 8: A^t←p¯t+W−p¯t−W+b⋅(2 1[yτ=1]−1) A_t← p_t+W- p_t-W+b·(2\, 1[y_τ=1]-1) 9: ct←0c_t← 0 10:end for 11:c←HDBSCAN(ft),ft=[q~t;βp¯t]\C_c\ (\f_t\),\;\;f_t=[\, q_t;\,β\, p_t\,] 12:for ata_t with A^t<0 A_t<0 do 13: Let cC_c be the HDBSCAN cluster assigned to ata_t 14: if ata_t is not an outlier and c+≠∅C_c^+≠ then 15: αi←softmax(A^i/κ)i∈c+ _i ( A_i/κ)_i _c^+ 16: at⋆←∑i∈c+αiaia_t ← _i _c^+ _i\,a_i 17: ct←1c_t← 1 18: end if 19:end for 20:// Phase 3: Adaptive redirection 21:for e=1,…,Ee=1,…,E do 22: θ←θ−η∇θℒθ←θ-η _θL using Eq. (14) 23:end for 24:return πθ _θ Appendix C Simulation Implementation Details Benchmark and base policy. We evaluate RedFlow on the four LIBERO suites [18]: Spatial, Object, Goal, and Long, each containing 10 tasks. We adopt π0 _0 [1] initialized from the π [3] checkpoint as the base VLA policy. The SFT policy is trained on a pruned expert set expD_exp containing 58 demonstrations for Spatial, Object, and Goal, and 208 demonstrations for Long. Offline RL data. RL fine-tuning uses =exp∪roll,D=D_exp _roll, (51) where rollD_roll contains 1,536 mixed-quality rollout trajectories per suite. Progress estimates are produced by the pretrained GRM, smoothed over temporal chunks, and combined with proprioceptive states for HDBSCAN-based corrective target assignment. All methods use the same training protocol on each task suite. Evaluation protocol. We report average success rates over 500 evaluation episodes per suite. The same evaluation protocol is used for all compared methods within each suite. Table 3: Hyperparameters for the single-iteration offline training pipeline of RedFlow on the LIBERO task suites. Spatial, Object, and Goal share the same configuration and SFT initialization. Long uses a longer episode horizon (480 vs. 240), which motivates the remaining changes: λsup _sup and λcor _cor are reduced to prevent accumulated suppression/correction signal from dominating the policy update over longer trajectories, while W and b are increased to stabilize progress estimation across more action steps. Bold values indicate settings changed for Long. Hyperparameter Spatial / Object / Goal Long Offline training schedule Policy update epochs 30 30 Action-level quality estimation Progress smoothing window, W 10 20 Outcome-bias coefficient, b 0.15 0.25 Soft-weight temperature, TwT_w 3.0 3.0 Attraction coefficient 1 1 Suppression strength, λsup _sup 0.3 0.1 Correction strength, λcor _cor 0.3 0.1 Corrective-target temperature, κ 1.0 1.0 Optimization Optimizer AdamW Learning rate 5×10−55× 10^-5 Adam coefficients, (β1,β2)( _1, _2) (0.9, 0.95) Weight decay 0.01 Gradient clipping 1.0 Micro / global batch size 128 / 2048 Rollout buffer and action chunks Training environments 64 64 Maximum episode length 240 480 Action-chunk length, K 5 5 Mixed-quality rollout trajectories 1,536 per suite Evaluation episodes 500 per suite Initialization and offline data Base policy initialization CSOGC_SOG CLongC_Long SFT demonstrations 58 208 • CSOG=RLinf-Pi0-SFT-Spatial-Object-GoalC_SOG= RLinf-Pi0-SFT-Spatial-Object-Goal; CLong=RLinf-SFT-Pi0-LIBERO-LongC_Long= RLinf-SFT-Pi0-LIBERO-Long. • W and b are defined in the action-level score A^t A_t; TwT_w is the temperature for the soft weight wtw_t; λsup _sup controls failure suppression; λcor _cor controls target-guided correction; κ is the temperature used to form the corrective target at⋆a_t . Table 4: HDBSCAN hyperparameters for progress–state-aware corrective target assignment in LIBERO. The four LIBERO suites share the same HDBSCAN configuration. Parameter Value Meaning min_cluster_size 15 Minimum HDBSCAN cluster size. min_samples 5 Minimum number of neighbors for a core point; larger values make clustering more conservative. progress_weight, β 5.0 Weight applied to the standardized progress dimension, making task progress more dominant in clustering. state_dim 7 Proprioceptive state dimension used for clustering, excluding the gripper dimension. Feature preprocessing StandardScaler Z-score normalization for the state and progress features before clustering. Centroid softmax temperature 1.0 Temperature for advantage-weighted softmax over positive samples in the same cluster. Hyperparameter choices for LIBERO. Tab. 3 summarizes the offline training configuration used across all four LIBERO suites, and Tab. 4 reports the HDBSCAN parameters used for corrective target assignment. The Spatial, Object, and Goal suites share an identical configuration, as their episode horizons and task structures are comparable. The Long suite, with twice the maximum episode length (480 vs. 240), requires several adjustments grounded in the role of each hyperparameter. Action quality estimation (W, b). Longer trajectories produce noisier progress signals at any single step, so we widen the smoothing window W from 10 to 20 and increase the outcome-bias coefficient b from 0.15 to 0.25 to place more weight on episode-level outcomes relative to instantaneous progress. Suppression strength (λsup _sup). The choice of λsup _sup reflects a key property of failed trajectories: not every action chunk in a failed episode is itself harmful, as failures often contain a substantial fraction of neutral chunks that simply do not contribute to task progress. An overly large λsup _sup would indiscriminately suppress these neutral chunks together with the genuinely harmful ones, over-concentrating the policy distribution and destabilizing training. We therefore keep λsup _sup moderate, and further reduce it from 0.3 to 0.1 on Long, where the suppression signal accumulates over more steps and this effect is more pronounced. Correction strength (λcor _cor). The corrective target at⋆a_t is not intended to replace the attraction term, which remains the primary driver of policy improvement; rather, λcor _cor provides a mild bias that shifts the action-sampling distribution toward higher-quality regions. A small value is therefore sufficient and preferred, and we use 0.3 for Spatial/Object/Goal and 0.1 for Long, again to account for the per-step accumulation over longer horizons. All other hyperparameters, including the optimizer configuration, batch sizes, rollout buffer size, and HDBSCAN corrective-target assignment settings, are shared across all four suites. Appendix D Real-robot Implementation Details Hardware and tasks. To evaluate RedFlow beyond simulation, we conduct real-world experiments on a dual-arm Agilex Cobot Magic robot equipped with three cameras, including one front-facing camera and two wrist-mounted cameras for visual input. We consider three tasks of increasing manipulation diversity: cloth folding, object sweeping, and table cleaning, covering dexterous bi-manual manipulation, tool-mediated interaction, and pick-and-place manipulation, respectively. Base policy and offline RL data. We initialize from the official π0 _0 base checkpoint provided by OpenPi [1]. For each real-world task, we obtain the base policy by fine-tuning on task-specific expert demonstrations for 50,000 steps. The expert demonstration counts are 600 for cloth folding, 200 for object sweeping, and 100 for table cleaning. For offline RL training, we collect 200, 100, and 100 rollouts from the initial policy for cloth folding, object sweeping, and table cleaning, respectively, yielding a mixed buffer of successes and failures. Progress estimation and corrective target assignment. For real-robot experiments, RoboDopamine/GRM progress predictions are computed with frame_interval=10. The HDBSCAN-based corrective target assignment uses min_cluster_size=50, min_samples=10, and progress weight β=5.0β=5.0. Table 5: Real-robot task instructions used for policy conditioning. Task Instruction Clothes Folding fold the cloth Object Sweeping Use the brush to sweep 5 small blocks into the dustpan one by one, then empty them into the basket beside the table and push down the brush on the table. Table Cleaning Pick up 3 blocks and put them in the yellow box one by one, then use the cleaning cloth to wipe up the milk spill. Table 6: Hyperparameters and setup for real-world experiments. All three tasks share the same π0 _0 LoRA backbone, adaptive redirection objective, optimizer, and distributed training configuration; task-specific items are listed per task. Hyperparameter Cloth Folding Object Sweeping Table Cleaning Hardware and control Robot platform Agilex Cobot Magic (dual-arm) Control frequency 100 Hz Cameras cam_high, cam_left_wrist, cam_right_wrist Camera model Intel RealSense D435 Camera frequency 30 Hz Action dimension, D 32 Action-chunk length, K 50 Backbone and initialization Vision-language backbone gemma_2b_lora Action expert backbone gemma_300m_lora Numerical precision bfloat16 Maximum language-token length 64 Base initialization gs://openpi-assets/checkpoints/pi0_base/params Task-finetuned initialization checkpoints/real_task/task/params Task fine-tuning steps 50,000 Normalization-statistics asset folding_cloth aloha_sweeping aloha_cleaning Progress estimation and corrective targeting GRM prediction frame interval 10 min_cluster_size 50 min_samples 10 Progress weight, β 5.0 Adaptive redirection objective Soft-weight temperature, TwT_w 3.0 λsup _sup 0.3 Adaptive margin scale for m 1.0 λcor _cor 0.3 Offline adaptation and optimization Policy update epochs 30 Batch size 32 Optimizer AdamW with cosine warmup Peak learning rate 2.0×10−52.0× 10^-5 Final learning rate 2.5×10−62.5× 10^-6 Warmup / cosine decay steps 1,000 / 30,000 Adam coefficients, (β1,β2)( _1, _2) (0.9, 0.95) Weight decay 1.0×10−101.0× 10^-10 Gradient-clipping norm 1.0 FSDP devices 8 Data and evaluation Expert demonstrations 600 200 100 Offline RL rollouts 200 100 100 Evaluation rollouts 100 100 100 Evaluation protocol. We evaluate each task over 100 rollouts, which provides substantially more reliable success-rate estimates than the 10–20 rollouts common in prior real-world VLA evaluations. Real-world episodes are not constrained by a fixed step limit; instead, each rollout terminates upon one of two conditions: (i) the task-specific success criterion is satisfied, or (i) the policy enters an unrecoverable failure state in which a manipulated object is dropped outside the workspace or in a configuration from which the policy cannot recover. Between rollouts, the scene is manually reset to a randomized initial configuration drawn from a task-specific distribution over object positions and orientations. Success criteria are defined per task as follows. Clothes folding: the robot folds the cloth into the target folded configuration without the garment falling off the table. Object sweeping: the robot uses the brush to sweep all 5 small blocks into the dustpan one by one, empties them into the basket beside the table, and pushes down the brush on the table. Table cleaning: the robot picks up all 3 blocks and places them into the yellow box one by one, then uses the cleaning cloth to wipe up the milk spill. A rollout that does not satisfy the corresponding success criterion before terminating is counted as a failure. Figure 8: Successful trajectory of cloth folding. Figure 9: Successful trajectory of object sweeping. Figure 10: Successful trajectory of table cleaning. Figure 11: Failed trajectory of cloth folding. Figure 12: Failed trajectory of object sweeping. Figure 13: Failed trajectory of table cleaning. Appendix E Diagnostics of Corrective-Target Assignment Table 7: Diagnostics of negative-chunk usage on LIBERO. All negative chunks are used for suppression; only negative chunks whose progress–state cluster contains positive chunks additionally receive target-guided correction. We report the fraction optimized with suppression only, the fraction optimized with suppression plus correction, and the mean proxy advantage of the positive chunks used to construct each target, A^⋆=∑i∈c+αiA^i A = _i _c^+ _i A_i. Suite Suppression only (%) Supp. + Corr. (%) Mean A^⋆ A Spatial 86.3 13.7 +0.216+0.216 Object 89.5 10.5 +0.274+0.274 Goal 89.3 10.7 +0.256+0.256 Avg 88.4 11.6 +0.243+0.243 Appendix F Limitations While RedFlow demonstrates strong improvements in both simulation and real-world manipulation tasks, it still has several limitations. First, its corrective target assignment depends on the quality of task-progress estimates and progress–state clustering; inaccurate progress signals or noisy clustering may lead to imperfect failure-to-success matching. Second, RedFlow can only provide constructive redirection for failures that have nearby positive support in the offline buffer. Failure modes that are novel, out-of-distribution, or lack successful counterparts are only suppressed rather than explicitly corrected. Finally, our experiments focus on LIBERO and three real-robot tasks with fixed embodiments and settings, and broader validation across more diverse robots, visual conditions, and long-horizon tasks remains an important direction for future work.