Paper deep dive
Online Evolution Strategy for Flow-Matching VLA Policies via Self-Supervised Trajectory Distribution Optimization
Gongxin Yao, Yongsheng Zhao, Jiayin Deng, Deng Liang, Han Gao, Lei Zhao, Baoping Cheng
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 10/4/2026, 3:19:16 AM
Summary
The paper introduces Online-ES, an online adaptation framework for Vision-Language-Action (VLA) models based on Flow Matching. It addresses the issue of ill-formed action trajectory distributions where successful and failed behaviors coexist. Online-ES uses Evolution Strategy (ES) to perform exploration directly in the action trajectory space by perturbing sampled trajectories. It derives a self-supervised Mean Squared Error (MSE) objective to transfer the evolution direction from trajectory space to model parameter space, proving it provides an unbiased estimator of the optimal direction. Additionally, it incorporates failure experiences as negative feedback to regularize the evolution, steering the policy away from failure regions. Experiments show it achieves policy improvement comparable to reinforcement fine-tuning without learning a value model.
Entities (8)
Relation Signals (7)
Online-ES → appliesto → Flow Matching
confidence 95% · online adaptation framework for Flow Matching VLAs
Online-ES → uses → Evolution Strategy (ES)
confidence 95% · Online-ES, an online adaptation framework for Flow Matching VLAs based on Evolution Strategy (ES)
Online-ES → derives → Self-Supervised MSE Objective
confidence 90% · derive a self-supervised MSE objective that transfers the evolution direction
Flow Matching → enables → Vision-Language-Action (VLA)
confidence 90% · Flow Matching enables VLA models to learn conditional action trajectory distributions
Online-ES → exploresin → Action Trajectory Space
confidence 90% · performs evolutionary exploration directly in the action trajectory space
Online-ES → incorporates → Failure-Aware Regularization
confidence 90% · incorporate failure experiences as negative feedback to regularize the evolution direction
Self-Supervised MSE Objective → transfersdirectionto → Model Parameter Space
confidence 85% · transfers the evolution direction from trajectory space into model parameter space
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Vision-Language-Action (VLA) models based on generative frameworks, such as Flow Matching, have recently achieved impressive performance in robotic manipulation. Unlike deterministic policies, Flow Matching enables VLA models to learn conditional action trajectory distributions, where latent noise vectors induce different actions under the same task scenario. However, we observe that these distributions are often ill-formed, with successful and failed behaviors coexisting while considerable probability mass remains in unfavorable regions. To this end, we propose Online-ES, an online adaptation framework for Flow Matching VLAs based on Evolution Strategy (ES), which refines the learned action trajectory distribution through interaction feedback. Instead of pruning the latent noise space, our method performs evolutionary exploration directly in the action trajectory space, where diverse trajectories generated by Flow Matching provide candidate solutions for adaptation. By perturbing sampled trajectories and evaluating their execution outcomes, we derive a self-supervised MSE objective that transfers the evolution direction from trajectory space into model parameter space. Mathematically, we prove that the proposed objective provides an unbiased estimator of the optimal evolution direction. Moreover, we also incorporate failure experiences as negative feedback to regularize the evolution direction, steering the policy away from previously explored failure regions. Experiments in both simulation and real-world environments demonstrate that Online-ES achieves policy improvement comparable to reinforcement fine-tuning, without learning a value model or computing advantages.
Tags
Links
- Source: https://arxiv.org/abs/2609.38855v1
- Canonical: https://arxiv.org/abs/2609.38855v1
Trouble viewing inline? Open PDF directly →
Full Text
46,344 characters extracted from source content.
Expand or collapse full text
Online Evolution Strategy for Flow-Matching VLA Policies via Self-Supervised Trajectory Distribution Optimization Gongxin Yao Yongsheng Zhao Jiayin Deng Deng Liang Han Gao Lei Zhao Baoping Cheng* †thanks: These authors are with the China Mobile (Hangzhou) Information Technology Co., Ltd., Hangzhou, 311121, China. (Baoping Cheng* is the corresponding author, email: chengbaoping@cmhi.chinamobile.com). Abstract Vision-Language-Action (VLA) models based on generative frameworks, such as Flow Matching, have recently achieved impressive performance in robotic manipulation. Unlike deterministic policies, Flow Matching enables VLA models to learn conditional action trajectory distributions, where latent noise vectors induce different actions under the same task scenario. However, we observe that these distributions are often ill-formed, with successful and failed behaviors coexisting while considerable probability mass remains in unfavorable regions. To this end, we propose Online-ES, an online adaptation framework for Flow Matching VLAs based on Evolution Strategy (ES), which refines the learned action trajectory distribution through interaction feedback. Instead of pruning the latent noise space, our method performs evolutionary exploration directly in the action trajectory space, where diverse trajectories generated by Flow Matching provide candidate solutions for adaptation. By perturbing sampled trajectories and evaluating their execution outcomes, we derive a self-supervised MSE objective that transfers the evolution direction from trajectory space into model parameter space. Mathematically, we prove that the proposed objective provides an unbiased estimator of the optimal evolution direction. Moreover, we also incorporate failure experiences as negative feedback to regularize the evolution direction, steering the policy away from previously explored failure regions. Experiments in both simulation and real-world environments demonstrate that Online-ES achieves policy improvement comparable to reinforcement fine-tuning, without learning a value model or computing advantages. I INTRODUCTION Vision-Language-Action (VLA) models [1] have recently emerged as a promising paradigm for general-purpose robot manipulation. Instead of learning semantic representations from scratch, a VLA model typically integrates a pretrained vision-language model [2] (VLM) with an action expert, where the VLM provides powerful grounding and reasoning capabilities to interpret visual observations and task instructions. The action expert is then trained on human teleoperation [3] datasets to map high-level VLM representations into low-level robot actions. In earlier studies [4, 5, 6], conventional neural network architectures, such as MLPs or Transformers, were commonly adopted as action experts to regress robot actions from observations. However, robot manipulation is inherently multimodal [7], where multiple feasible action trajectories may achieve the same task objective within the same scenario. Consequently, these action experts tend to produce a single deterministic trajectory, collapsing diverse solutions into averaged behaviors. Flow Matching is a generative framework [8] that learns a continuous velocity field to transport samples from noise distribution to the target data distribution through an ordinary differential equation (ODE). It has achieved remarkable success in image and video generation [9, 10], demonstrating the ability to produce high-quality and diverse content. Recently, an increasing number of VLA models [11, 12, 13] have adopted Flow Matching as a new paradigm for action generation. Unlike deterministic action prediction, Flow Matching VLA models implicitly learn an action trajectory distribution conditioned on observation, where random latent noise vectors serve as the initial states of action generation, enabling diverse trajectories to be sampled from the learned distribution. Although such intrinsic stochasticity provides behavioral diversity, it also introduces variability in execution results. Specifically, the generation process may occasionally sample an unfavorable latent noise vector, resulting in task failure. Meanwhile, other latent noise vectors may generate successful trajectories for the same task. This leads to our key insight: task failure in a single rollout may arise from the ill-formed action trajectory distribution, rather than the policy’s inability to solve the task. Fig. 1: Flow Matching VLAs learn a conditional velocity field that maps noise distributions to action trajectory distributions. For each task scenario, we repeatedly sample different latent noise vectors to generate diverse action trajectories and execute them. Experiences from both successful and failed trajectories are then used to adapt the model, progressively reshaping the action trajectory distribution toward regions with more favorable behaviors. Motivated by the above analysis, an intuitive idea is to prune the sampling space of the latent noise vector [14], thereby steering future generations away from unfavorable regions. However, a long-horizon action trajectory typically consists of multiple action chunks [15], each generated based on the observations resulting from its preceding chunks. Although the latent noise vector for each action chunk is sampled independently, their individual effects on the final execution result are difficult to disentangle due to the Markovian dependency. Moreover, noise-space pruning is inherently task-specific, as the unfavorable regions of the noise space vary substantially across task scenarios, making pruning rules difficult to generalize. Instead, we seek to adapt the model online according to the execution feedback as illustrated in Fig. 1, thereby continuously refining the action trajectory distribution. In this paper, we propose Online-ES, a simple online adaptation framework for Flow Matching VLAs based on Evolution Strategy [16]. This framework exploits the stochasticity of Flow Matching to generate diverse candidate trajectories for evolutionary exploration in the action trajectory space. Specifically, for each task scenario, we repeatedly reset the environment and sample different latent noise vectors to obtain a set of action trajectories, which provide a Monte Carlo approximation of the underlying action trajectory distribution. Each sampled action trajectory then undergoes local exploration through Gaussian perturbations and is evaluated through interaction with the environment. Based on the feedback-weighted perturbation responses, we derive a self-supervised MSE objective that transfers the evolution direction from trajectory space into parameter space for model adaptation. Mathematically, we prove that the proposed objective provides an unbiased estimator of the optimal evolution direction, thereby guiding the trajectory distribution toward regions with higher expected feedback. Furthermore, by incorporating complementary negative feedback from failed experiences, we introduce an implicit repulsive force that discourages the policy from revisiting previously explored failure regions. Experiments in simulation and real-world environments demonstrate that Online-ES can achieve policy improvement comparable to reinforcement fine-tuning, without learning a value model [17] or computing complex advantages [18, 19]. In summary, our main contributions are: • An online adaptation framework for Flow Matching VLAs based on Evolution Strategy, which performs evolutionary exploration in action trajectory space to refine the ill-formed trajectory distribution. • A self-supervised paradigm to optimize Flow Matching VLAs, which learns from self-generated interaction trajectories without learning an additional value model or computing complex advantages. • A failure-aware regularization mechanism that incorporates failure experiences as complementary negative feedback, steering the evolution direction away from previously explored failure regions. I Related Work I-A Flow Matching for Generative Modeling Flow Matching [8] has emerged as a prominent framework for efficient generative modeling. Unlike diffusion models [20], which rely on multi-step reverse denoising to generate samples from noise distributions, Flow Matching learns a time-dependent vector field that transports samples along a continuous probability path through an ordinary differential equation (ODE). Note that the initial noise vector specifies the starting point of ODE integration, and different initializations lead to distinct transport paths under the learned vector field, resulting in diverse samples from the target distribution [8]. Thus, this formulation provides a unified and efficient framework for modeling complex multimodal distributions, enabling remarkable advances in text [21], image [9], and video [10] generation. I-B Vision-Language-Action Models VLA models have emerged as a new paradigm for robotic policy learning by integrating pretrained vision-language backbones with action experts [1]. In real-world manipulation, a single task often admits multiple feasible solutions, e.g., the same object can be grasped from different poses or approached through different paths. There is an increasing trend toward adopting generative modeling frameworks [22, 23] to learn such multimodal behaviors. Notably, recent studies, including π0.5 _0.5 [11], GR00T [12], and some World Action Models [24, 25], have explored Flow Matching-based action generation by learning vector fields conditioned on visual observations and language instructions. Moreover, recent studies have introduced reinforcement learning to further improve VLA policies beyond supervised fine-tuning, including VLA-RL [17] with process-level value learning, πRL _RL [18] with group-relative advantages, and VLA-RFT [19] with world-model-based verified rewards. In contrast, our method exploits the inherent action trajectory diversity of Flow Matching VLAs for evolutionary exploration, achieving similar policy improvement without explicit value modeling or complex advantage computation. I-C Evolution Strategy Evolution Strategy (ES) is a class of gradient-free optimization methods inspired by biological evolution principles [26], and has been widely applied to continuous black-box optimization problems. Natural Evolution Strategies [16] formulate evolutionary optimization from a probabilistic perspective by optimizing the expected fitness of a search distribution, providing a principled way to estimate optimization directions through stochastic perturbations. It has also been extended to optimize neural networks by applying stochastic perturbations to the parameters, demonstrating that models can be trained without backpropagation [27]. Recent advances in ES have further improved its effectiveness for black-box optimization by developing adaptive sampling and distribution update strategies, which enhance exploration efficiency and convergence performance [28, 29, 30]. In this work, we will extend ES to the action trajectory space of Flow Matching VLA policy for online adaptation. I Preliminaries I-A Problem Formulation Given a Flow-Matching VLA policy πθ(⋅) _θ(·) that learns a velocity field conditioned on the observation o and language instruction l, an action chunk is generated by integrating the following ordinary differential equation over t∈[0,1]t∈[0,1]: dtdt=πθ(t,t,,),t=0=,∼(0,I), aligned d A_tdt&= _θ( A_t,t, o, l),\\ A_t=0&= z, z (0,I), aligned (1) where the result at t=1t=1 is the final action chunk A. Notably, the latent noise vector z specifies the initial condition of the ODE integration. Different realizations can thus lead to different actions under the same observation-language input: ≠′⟹≠′. z≠ z A≠ A . (2) Fig. 2: Action trajectories generated by a pretrained flow-matching VLA policy from different latent noise vectors within the same task scenario. In practice, the policy interacts with the environment to sequentially generate action chunks, with each chunk affecting the subsequent observations and action generations as: (0)→(0)→(1)→(1)→(2)→(2)→⋯ o^(0)→ A^(0)→ o^(1)→ A^(1)→ o^(2)→ A^(2)→·s (3) where the superscripts denote the interaction steps and all chunks form a long-horizon action trajectory τ. Although the variations induced by different latent noise vectors may be slight at early stages, this feedback loop allows the effects to accumulate over time, resulting in different trajectories over long horizons. For simplicity, we summarize this closed-loop generation process for a given task scenario S as: τ=πθ(,),∼(0,I), τ= _θ(S, z), z (0,I), (4) which induces a conditional action trajectory distribution: τ∼pθ(τ|), τ p_θ(τ|S), (5) where each realization of z corresponds to a trajectory sampled from this distribution. Fig. 2 shows the results of a representative experiment: Under the same task scenario, different latent noise vectors produce diverse action trajectories, among which both successful and failed behaviors exist. This suggests that task failure in a single rollout may result from sampling an unfavorable latent noise vector, rather than the policy’s inability to solve the task. Essentially, the coexistence of success and failure indicates that the action trajectory distribution in Eq. (5) is ill-formed, motivating us to refine it online. I-B Evolution Strategy Fundamentals Evolution Strategy (ES) is a population-based optimization method inspired by natural selection, which iteratively explores the solution space through random perturbations and fitness-based selection. Given an objective function J(θ)J(θ), ES generates a population of perturbed candidates around the current solution θ to explore its local neighborhood: θi=θ+ϵi,ϵi∼(0,σ2I), _i=θ+ _i\,,\; _i (0,σ^2I), (6) where σ controls the perturbation scale. Then, the fitness of each candidate is computed as: J(θi)=J(θ+ϵi). J( _i)=J(θ+ _i). (7) Here, J(θ)J(θ) is defined as a maximization objective, where higher values indicate better solutions. The optimization direction is estimated by aggregating the perturbations weighted by their fitness as follows: ∇θ=1nσ2∑i=1nJ(θi)⋅ϵi. _θ= 1nσ^2Σ^n_i=1J( _i)· _i. (8) The solution is then updated as: θ←θ+α⋅∇θ, θ←θ+α· _θ, (9) where α denotes the updating step size. By selecting perturbation directions associated with higher fitness, ES enables gradient-free optimization without explicit derivatives of the objective function. This strategy has been successfully extended to deep neural networks [27] as a scalable alternative to reinforcement learning. IV METHODOLOGY IV-A Extending Evolution Strategy to Action Trajectory Space The goal of online adaptation is to refine the underlying trajectory distribution in Eq. (5): maxθτ∼pθ(τ∣)[J(τ)], _θ\;E_τ p_θ(τ ) [J(τ) ], (10) where J(τ)J(τ) denotes the execution feedback from the environment. In robotic manipulation tasks, the feedback is typically formulated as a binary success indicator: J(τ)=1,if the task succeeds by executing τ,0,otherwise, J(τ)= cases1,&if the task succeeds by executing τ,\\ 0,&otherwise, cases (11) which makes Eq. (10) equivalent to maximizing the average success rate. Following prior work [27], a straightforward way to apply ES here is to perturb the policy parameters, thereby generating a population of candidate policies: πθ(⋅)→πθ+ϵ(⋅),ϵ∼(0,σ2I). _θ(·)→ _θ+ε(·),\;\;ε (0,σ^2I). (12) However, the massive number of parameters in VLA models makes such parameter-space exploration extremely inefficient. More critically, the deep architecture may amplify even small parameter perturbations, potentially leading to action trajectories that are physically infeasible or unsafe to execute. Therefore, we instead perform stochastic exploration directly in the action trajectory space as: pθ(τ∣)→pθ(τ+ϵ∣),ϵ∼(0,σ2I). p_θ(τ )→ p_θ(τ+ε ),\;\;ε (0,σ^2I). (13) By controlling the perturbation magnitude σ, we can preserve the base behavior of the pretrained VLA model while performing stochastic exploration in its local neighborhood. The beneficial explorations are then transferred to parameter space, thereby refining the action trajectory distribution. IV-B Evolution Direction of Action Trajectory Distribution Since an individual action trajectory only provides a limited view of the distribution implicitly represented by model parameters, we estimate its overall evolution direction through a sample-then-explore scheme. Specifically, given a resettable task scenario S, we sample multiple latent noise vectors to generate a set of action trajectories as: Ω=τi∣τi=πθ(,i),i∼(0,I)i=1N. = \ _i _i= _θ(S, z_i), z_i (0,I) \_i=1^N. (14) It provides a Monte Carlo approximation of the underlying action trajectory distribution, as illustrated in Fig. 2. For each action trajectory in Ω , we explore its local neighborhood by applying Gaussian perturbations with magnitude σ as: τ^i=τi+ϵi,ϵi∼(0,σ2I). τ_i= _i+ _i, _i (0,σ^2I). (15) Following the principle of ES, the evolution objective is to maximize the execution feedback of perturbed action trajectories. Mathematically, for any trajectory τ, we define its expected feedback under stochastic perturbations as: (τ) (τ) =ϵ[J(τ+ϵ)] =E_ε[J(τ+ε)] (16) =∫J(τ+ϵ)⋅p(ϵ)dϵ. = J(τ+ε)· p(ε)\,dε. Therefore, the gradient of (τ)J(τ) with respect to τ provides the steepest ascent direction as: ∇τ(τ) _τJ(τ) =∇τ∫J(τ+ϵ)⋅p(ϵ)dϵ = _τ J(τ+ε)· p(ε)\,dε =∫J(x)⋅∇τp(x−τ)dx(x=τ+ϵ) = J(x)· _τp(x-τ)\,dx (x=τ+ε) =∫J(x)⋅p(x−τ)⋅∇τlogp(x−τ)x = J(x)· p(x-τ)· _τ p(x-τ)\,dx =x∼p(x−τ)[J(x)⋅∇τlogp(x−τ)]. =E_x p(x-τ) [J(x)· _τ p(x-τ) ]. (17) Since the perturbation noise follows a Gaussian distribution in Eq. (15), the derivative term in Eq. (17) is: ∇τlogp(x−τ) _τ p(x-τ) =∇τlog(12πσexp(−‖x−τ‖22σ2)) = _τ ( 1 2πσ (- \|x-τ\|^22σ^2 ) ) =∇τ(−‖x−τ‖22σ2+C) = _τ (- \|x-τ\|^22σ^2+C ) =x−τσ2=ϵσ2, = x-τσ^2= εσ^2, (18) Note that the expectation over x∼p(x−τ)x p(x-τ) in Eq. (17) is equivalent to the expectation over the perturbation noise ϵε. Consequently, substituting Eq. (18) into Eq. (17) yields: ∇τ(τ)=1σ2ϵ[J(τ+ϵ)ϵ]. _τJ(τ)= 1σ^2E_ε [J(τ+ε)ε ]. (19) For each action trajectory, this result demonstrates that the feedback-weighted perturbations provide an unbiased estimate of the evolution direction. Then, the overall evolution direction of the action trajectory distribution is given as: τ∼pθ(τ∣)[∇τ(τ)] _τ p_θ(τ ) [ _τJ(τ) ] =τ∼pθ(τ∣)[1σ2ϵ[J(τ+ϵ)ϵ]] =E_τ p_θ(τ ) [ 1σ^2E_ε [J(τ+ε)ε ] ] =1σ2τ∼pθ(τ∣),ϵ[J(τ+ϵ)ϵ]. = 1σ^2E_τ p_θ(τ ),\,ε [J(τ+ε)ε ]. (20) IV-C Self-Supervision for Parameter Adaptation Although Eq. (20) provides the evolution direction in the trajectory space, it must be transferred to the parameter space for online model adaptation. To bridge the two spaces, we construct a self-supervised MSE loss based on the sampled action trajectories in Eq. (14) and their perturbed variants in Eq. (15), weighted by the execution feedback as: ℒ=1|Ω|∑τi∈ΩJ(τ^i)⋅‖τ^i−τi‖22, = 1| | _ _i∈ J( τ_i)·\| τ_i- _i\|_2^2, (21) where J(τ^i)J( τ_i) denotes the execution feedback of τ^i τ_i. Although τ^i τ_i is generated from τi _i, we detach τ^i τ_i from the computation graph during implementation. Consequently, the gradients are back-propagated only through τi _i to update the model parameters. The resulting gradient of ℒL with respect to the model parameters θ is given by: ∇θℒ _θL =1|Ω|∑τi∈ΩJ(τ^i)⋅∇θ‖τ^i−τi‖22 = 1| | _ _i∈ J( τ_i)· _θ\| τ_i- _i\|_2^2 =−2|Ω|∑τi∈ΩJ(τ^i)⋅(τ^i−τi)⋅∇θτi =- 2| | _ _i∈ J( τ_i)·( τ_i- _i)· _θ _i =−2|Ω|∑τi∈ΩJ(τ^i)⋅ϵi⋅∇θτi, =- 2| | _ _i∈ J( τ_i)· _i· _θ _i, (22) where ∇θτi=∇θπθ(,i) _θ _i= _θ\, _θ(S, z_i). For each action trajectory τ, taking expectation over its perturbation noise ϵε gives: [∇θℒ] [ _θL ] =−2⋅∇θτ⋅[J(τ^)⋅ϵ] =-2· _θτ·E [J( τ)·ε ] =−2⋅∇θτ⋅[J(τ+ϵ)⋅ϵ] =-2· _θτ·E [J(τ+ε)·ε ] ∝−∇θτ⋅∇τ(τ). - _θτ· _τJ(τ). (23) This result shows that the MSE loss projects the evolution direction in Eq. (19) into the parameter space through the model Jacobian ∇θτ _θτ. The negative sign will be canceled by the gradient descent rule used to minimize the loss. Similarly, the expectation over all samples in Ω also projects the overall evolution direction in Eq. (20) into the parameter space. IV-D Failure-Aware Regularization Although successful action trajectories provide positive guidance for model adaptation, failed trajectories also contain valuable information that has not yet been exploited. We follow the same intuition as navigating a maze: once a route proves to be a dead end, it should not be revisited. Likewise, a robot should learn to avoid generating action trajectories that have been proven to fail. To this end, we modify the execution feedback in Eq. (11) as: J(τ)=1,if the task succeeds by executing τ,−1,otherwise. J(τ)= cases1,&if the task succeeds by executing τ,\\ -1,&otherwise. cases (24) Thus, the gradient of ℒL in Eq. (22) can be regularized as: ∇θℒ _θL =−2|Ω|(∑τi∈ΩSϵi∇θτi−∑τj∈ΩFϵj∇θτj), =- 2| | ( _ _i∈ _S _i _θ _i- _ _j∈ _F _j _θ _j ), (25) where Ω=ΩS∪ΩF = _S∪ _F, with ΩS=τi∈Ω|J(τi)=1 _S=\ _i∈ |J( _i)=1\ and ΩF=τi∈Ω|J(τi)=−1 _F=\ _i∈ |J( _i)=-1\ denoting the subsets of successful and failed trajectories, respectively. As illustrated in Fig. 3, the joint effect of the two gradient terms can be interpreted as follows: the successful term provides an attractive signal that pulls the trajectory distribution toward high-feedback regions, while the failure term provides a complementary repulsive signal that pushes the distribution away from previously explored failure regions. Fig. 3: Illustration of how successful and failed experiences jointly reshape the action trajectory distribution. V Experiment V-A Implementation Details Online Adaptation Procedure. Algorithm 1 summarizes the proposed online adaptation framework for a single task scenario. It can be extended to multi-task scenarios by parallelizing across multiple GPUs, where each GPU collects action trajectories for one task and synchronizes multi-GPU gradients before policy updates. In real-world robot execution, the perturbed action trajectories may induce joint oscillations due to high-frequency variations, which can occur even in the original policy outputs. Therefore, we apply mathematical tools, such as quintic polynomial fitting or spline interpolation, to improve the smoothness of action trajectories before execution. This smoothing constraint does not affect the derivation in Sec. IV-B, since J(τ^)J( τ) serves as an independent weighting term in the gradient estimation. Moreover, we control the ratio between the successful and failed trajectories in Eq. (25), with |ΩS|>2|ΩF|| _S|>2| _F| and |Ω|=10| |=10. Empirically, this constraint allows the positive evolution direction to dominate the gradient estimation, leading to a more stable online adaptation process. Algorithm 1 Online-ES Adaptation Framework Input: Interactive Environment EnvEnv, Pretrained Flow-Matching VLA policy πθ _θ; Initialize rollout buffer Ω ; while online adaptation is not terminated do Reset EnvEnv to the same initial state; Initialize action trajectory buffer (τ, τ τ); while the task is not terminated in EnvEnv do Get observations from EnvEnv; Generate a random latent noise vector; Generate an action chunk using Eq. (1); Perturb the action chunk using Eq. (15); Execute the perturbed action chunk in EnvEnv; Append the raw / perturbed action chunks to (τ, τ τ); Append (τ, τ τ) with its execution feedback J(τ^)J( τ) to Ω ; if enough trajectories are collected in Ω then Compute the self-supervised loss using Eq. (21); Update πθ _θ using the gradient in Eq. (25); Clear rollout buffer Ω ; Baseline Model. We adopt GR00T N1.5 [12] as the baseline model, augmented with depth-based 3D features following GeoVLA [31]. This baseline model consists of a vision encoder, a depth encoder, a language encoder, and a DiT-based action expert. The features extracted from the first three modules serve as conditional inputs of the action expert for action generation. During the online adaptation stage, we only finetune the parameters of the action expert, while keeping other modules frozen. Hardware Resources. In simulation experiments, we utilize 8 H100 GPUs for parallel model adaptation, with each GPU responsible for one task scenario. For real-world experiments, we build the task scenario with an AgiBot G1 robot [32], which consists of dual arms with 14-DoF joints and two parallel-jaw grippers. Since only one task scenario is involved in the real-world setting, the model adaptation uses only a single H100 GPU. V-B Simulation Results We conducted simulation experiments on the LIBERO benchmark [33], which consists of 4 task suites with 10 tasks per suite. We first fine-tune the baseline model on the official human demonstration dataset using offline supervised learning (Offline-SFT) for 30000 training steps. The finetuned model then interacts with the benchmark environments and is continuously refined using our proposed Online-ES. Fig. 4: The curve of single-task adaptation with fixed scene initialization. Single-Task Adaptation with Fixed Scene Initialization. In this setting, we randomly select one task from the LIBERO benchmark and perform online adaptation under a fixed scene configuration (i.e., same object placement). The perturbation magnitude σ in Eq. (15) is set to 0.03. Fig. 4 shows the success rate curves on the 7th task of the LIBERO Spatial benchmark. For a clear comparison, we also use the Offline-SFT baseline to generate action trajectories and compute the success rates synchronously. As we can see, Online-ES framework improves the average success rate to nearly 100% after 300 adaptation steps. Fig. 5: The curves of single-task adaptation with random scene initialization. TABLE I: The results of single-task adaptation with random scene initialization on LIBERO benchmark. Spatial 1st1st Task 7th7th Task 8th8th Task 9th9th Task Baseline 0.922 0.934 0.916 0.902 Online-ES 0.994 ↑ 0.992 ↑ 0.996 ↑ 0.970 ↑ Object 3rd3rd Task 4th4th Task 7th7th Task 9th9th Task Baseline 0.976 0.952 0.980 0.974 Online-ES 0.994 ↑ 0.996 ↑ 0.996 ↑ 0.998 ↑ Goal 3rd3rd Task 5th5th Task 6th6th Task 9th9th Task Baseline 0.912 0.916 0.960 0.934 Online-ES 0.960 ↑ 0.988 ↑ 0.992 ↑ 0.978 ↑ Long 4th4th Task 6th6th Task 8th8th Task 9th9th Task Baseline 0.912 0.860 0.784 0.902 Online-ES 0.970 ↑ 0.984 ↑ 0.956 ↑ 0.968 ↑ Single-Task Adaptation with Random Scene Initialization. In this setting, we randomly select one task from the LIBERO benchmark and perform adaptation across different scene configurations (i.e., 50 object placements). The perturbation magnitude σ is set to 0.03. Fig. 5 shows the success-rate curves for one representative task from each of the four task suites. Although the success rate consistently improves for all tasks, it increases at a slower pace compared with Fig. 4, and the final performance remains below 100%. This degradation may result from the variations in visual observations caused by different object placements, which can lead to conflicting gradients. To further validate the effectiveness, we have selected more tasks to perform online adaptation. Table I reports the success rates. Fig. 6: The curves of multi-task adaptation with random scene initialization. TABLE I: The results of multi-task adaptation with random scene initialization on LIBERO benchmark. * represents the best. Spatial Object Goal Long Average OpenVLA[34] 0.847 0.884 0.792 0.537 0.765 VLA-RL[17] 0.902 0.918 0.822 0.598 0.810 VLA-RFT [19] 0.944 0.944 0.954 0.802 0.911 π0.5 _0.5 [11] 0.988 0.982 0.980 0.924 0.969 πRL _RL[18] 0.996* 1.000* 0.996* 0.940* 0.983* Baseline 0.938 0.978 0.958 0.918 0.948 Online-ES 0.978 ↑ 0.996 ↑ 0.980 ↑ 0.936 ↑ 0.972 ↑ Fig. 7: Real-world experimental setup. (a) The task scenario in which the robot attempts to grasp the pink bowl. (b) The action trajectory distribution visualized by overlaying the arm motions (see the ghosted region) from multiple rollouts. (c) Three execution outcomes with top views shown above and side views shown below. Multi-Task Adaptation with Random Scene Initializations. In this setting, we perform joint online adaptation across all tasks within each task suite (10 tasks × 50 different object placements) of the LIBERO benchmark. During online adaptation, 8 GPUs perform rollouts in parallel, with each GPU handling one task. Accounting for the increased difficulty, we adopt an annealing strategy for the perturbation magnitude σ, linearly decreasing it from 0.3 to 0.03 over the first 2,500 steps. Fig. 6 shows the success rate curves for the four task suites. The larger exploration range initially leads to substantially lower success rates than the baseline, but performance gradually surpasses the baseline as the exploration range decreases. Table I reports the quantitative results, with VLA-RL based on OpenVLA and πRL _RL based on π0.5 _0.5 for reinforcement fine-tuning. Despite not achieving the highest success rate, Online-ES achieves competitive policy improvement with its simpler adaptation scheme. Fig. 8: Real-World Action Trajectory Perturbation. (a–c) Three examples of perturbed trajectories (gray) and refitted trajectories (cyan) generated from the same original action trajectory (magenta). (d) The local exploration region around the raw action trajectory formed by multiple perturbations. TABLE I: The Quantitative Results of Real-World Online Adaptation, which are evaluated over 100 independent trials respectively. Failure Invalid Contact Success Baseline 27 / 100 41 / 100 32 / 100 Online-ES (50 steps) 22 / 100 ↓ 42 / 100 ↑ 36 / 100 ↑ Online-ES (100 steps) 19 / 100 ↓ 37 / 100 ↓ 44 / 100 ↑ Online-ES (150 steps) 12 / 100 ↓ 34 / 100 ↓ 54 / 100 ↑ V-C Real-World Results Fig. 7 (a) shows the real-world experimental setup for the tabletop clearing task. For ease of environment resetting, we select the subtask of grasping the pink bowl as the evaluation benchmark. We collect 300 human teleoperation demonstrations and train the policy offline for 30,000 steps to obtain a baseline checkpoint. By projecting action trajectories generated from multiple rollouts onto a single image, we can observe a coarse outline of the underlying action trajectory distribution as shown in Fig. 7 (b). To evaluate the model performance, we categorize the grasping outcomes into three classes, as illustrated in Fig. 7 (c). Failure denotes cases where the gripper completely misses the bowl. Invalid Contact refers to cases where the gripper reaches the bowl rim with an improper grasping pose. Although these cases also fail to complete the task, they are considered closer to successful grasping than Failure. Finally, Success denotes the cases where the gripper has a proper grasping pose, with the bowl rim positioned between the two fingers. In real-world experiments, robot motors are highly sensitive to high-frequency input variations, and perturbing the action trajectory according to Eq. (15) can induce severe joint oscillations. We follow the post-processing strategy [35] to refit the perturbed action trajectories using a high-order polynomial and then execute them. As illustrated in Fig. 8, this procedure transforms the independent noise added to individual actions into a smooth trajectory-level perturbation over a longer horizon, thereby still forming a local exploration region around the original action trajectory. We adapted the model online for 150 steps. The perturbation magnitude σ is set to 0.05 for the AgiBot G1 robot. At each step, we fill the buffer Ω with 10 action trajectories and compute their averaged gradients to update the model parameters. Note that the ratio of successful to failed trajectories in Ω is maintained at 8:2. When the baseline checkpoint performs poorly, we have to run more than 10 trials to collect action trajectories. Tab. I reports the quantitative results, where each row is evaluated over 100 rollouts. We can observe clear decreasing trends for both Failure cases and Invalid Contact cases, demonstrating that the action trajectory distribution of the policy has been progressively refined. Fig. 9: The success rate curves with different buffer sizes |Ω| . Fig. 10: The success rate curves with different perturbation magnitudes σ. V-D Ablation Studies The size of action trajectory buffer Ω in Eq. (14). This hyperparameter controls how well each online-adaptation step captures the current action trajectory distribution, with a larger |Ω| providing a more comprehensive estimate. We conducted this ablation study on the 7th task of the LIBERO Spatial benchmark. Fig. 10 shows the success rate curves for |Ω| set to 1, 5, and 10. As |Ω| increases, the success rate exhibits a smoother and more stable upward trend. The effect of perturbation magnitude σ in Eq. (15). This hyperparameter controls the exploration range, with σ specifying the standard deviation of the injected noise. We again conducted this ablation study on the 7th task of the LIBERO Spatial benchmark. Fig. 10 shows the success rate curves for a fixed σ=0.03σ=0.03 and a linear annealing schedule that decays σ from 0.3 to 0.03 over 2500 adaptation steps. It can be observed that both curves exhibit a clear upward trend. In the early stage, the annealed setting achieves substantially lower success rates due to its larger exploration range. As σ decreases, its performance catches up with and eventually surpasses that of the fixed-σ setting. This phenomenon suggests that appropriately broad exploration can capture valuable information that may be missed by a narrow exploration range. Fig. 11: The success rate curves with a hybrid buffer of successful / failed trajectories versus a success-only buffer. The effect of failure experiences in Eq. (25). The gradients derived from failure experiences can regularize the evolution direction, discouraging the model from revisiting action trajectories that have previously led to failure. To verify this effect, we conducted this ablation study across all tasks of the LIBERO Spatial benchmark. Fig. 11 shows the average success rate curves for online adaptation with successful experiences only and mixed experiences. The results demonstrate that failure-based regularization can accelerate the convergence of success rate. VI CONCLUSION In this paper, we propose an Evolution Strategy-based online adaptation framework for Flow Matching VLAs, which refines the learned action trajectory distribution through execution feedback. Our key observation is that task failures may arise from an ill-formed trajectory distribution rather than the lack of feasible solutions. By exploiting the stochasticity of Flow Matching, our method performs trajectory-space exploration and adapts model parameters through a self-supervised objective with failure-aware regularization. Extensive experiments in simulation and real-world robotic manipulation tasks demonstrate that our framework effectively improves the performance of Flow Matching VLAs without additional human demonstrations. Future work will focus on improving the efficiency of utilizing the interaction experiences for online adaptation. References [1] Y. Ma, Z. Song, Y. Zhuang, J. Hao, and I. King (2026) A survey on vision–language–action models for embodied ai. IEEE Transactions on Neural Networks and Learning Systems. Cited by: §I, §I-B. [2] J. Zhang, J. Huang, S. Jin, and S. Lu (2024) Vision-language models for vision tasks: a survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 46 (8), p. 5625–5644. Cited by: §I. [3] W. Si, N. Wang, and C. Yang (2021) A review on manipulation skill acquisition through teleoperation-based learning from demonstration. Cognitive Computation and Systems 3 (1), p. 1–16. Cited by: §I. [4] M. Shridhar, L. Manuelli, and D. Fox (2022) CLIPort: what and where pathways for robotic manipulation. In Conference on Robot Learning, p. 894–906. Cited by: §I. [5] B. Zitkovich, T. Yu, S. Xu, et al. (2023) RT-2: vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, p. 2165–2183. Cited by: §I. [6] C. Yin and Q. Zhang (2023) A multi-modal framework for robots to learn manipulation tasks from human demonstrations. Journal of Intelligent & Robotic Systems 107 (4), p. 56. Cited by: §I. [7] X. Zhai, Q. Zhao, Q. Yu, and C. Hao (2025) VFP: variational flow-matching policy for multi-modal robot manipulation. arXiv preprint arXiv:2508.01622. Cited by: §I. [8] Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le (2023) Flow matching for generative modeling. In International Conference on Learning Representations, Cited by: §I, §I-A. [9] S. Xiao, Y. Wang, J. Zhou, et al. (2025) OmniGen: unified image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 13294–13304. Cited by: §I, §I-A. [10] A. Kodaira, T. Hou, J. Hou, M. Georgopoulos, F. Juefei-Xu, M. Tomizuka, and Y. Zhao (2026) StreamDiT: real-time streaming text-to-video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 29200–29210. Cited by: §I, §I-A. [11] K. Black, N. Brown, J. Darpinian, et al. (2025) π0.5 _0.5: A vision-language-action model with open-world generalization. In Proceedings of The 9th Conference on Robot Learning, Vol. 305, p. 17–40. Cited by: §I, §I-B, TABLE I. [12] J. Bjorck, F. Castañeda, N. Cherniadev, et al. (2025) Gr00T N1: an open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503.14734. Cited by: §I, §I-B, §V-A. [13] W. Wu, F. Wang, F. Lu, H. Sun, S. Liu, Y. Wang, Y. Yan, Y. Wang, S. Ma, X. Wang, et al. (2026) From foundation to application: improving vla models in practice. arXiv preprint arXiv:2607.06403. Cited by: §I. [14] G. Stoica, V. Ramanujan, X. Fan, et al. (2025) Contrastive flow matching. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 1185–1194. Cited by: §I. [15] T. Zhao, V. Kumar, S. Levine, and C. Finn (2023) Learning fine-grained bimanual manipulation with low-cost hardware. Robotics: Science and Systems XIX. Cited by: §I. [16] D. Wierstra, T. Schaul, T. Glasmachers, Y. Sun, J. Peters, and J. Schmidhuber (2014) Natural evolution strategies. The Journal of Machine Learning Research 15 (1), p. 949–980. Cited by: §I, §I-C. [17] G. Lu, W. Guo, C. Zhang, et al. (2025) VLA-RL: towards masterful and general robotic manipulation with scalable reinforcement learning. arXiv preprint arXiv:2505.18719. Cited by: §I, §I-B, TABLE I. [18] K. Chen, Z. Liu, T. Zhang, et al. (2025) πRL _RL: Online RL fine-tuning for flow-based vision-language-action models. arXiv preprint arXiv:2510.25889 8. Cited by: §I, §I-B, TABLE I. [19] H. Li, P. Ding, R. Suo, Y. Wang, Z. Ge, D. Zang, K. Yu, M. Sun, H. Zhang, D. Wang, et al. (2025) VLA-RFT: vision-language-action reinforcement fine-tuning with verified rewards in world simulators. arXiv preprint arXiv:2510.00406. Cited by: §I, §I-B, TABLE I. [20] F. Croitoru, V. Hondru, R. T. Ionescu, and M. Shah (2023) Diffusion models in vision: a survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 45 (9), p. 10850–10869. Cited by: §I-A. [21] V. Hu, D. Wu, Y. Asano, et al. (2024) Flow matching for conditional text generation in a few sampling steps. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, p. 380–392. Cited by: §I-A. [22] E. Chisari, N. Heppert, M. Argus, T. Welschehold, T. Brox, and A. Valada (2025) Learning robotic manipulation policies from point clouds with conditional flow matching. In Proceedings of The 8th Conference on Robot Learning, Vol. 270, p. 982–993. Cited by: §I-B. [23] C. Chi, Z. Xu, S. Feng, E. Cousineau, Y. Du, B. Burchfiel, R. Tedrake, and S. Song (2025) Diffusion Policy: visuomotor policy learning via action diffusion. The International Journal of Robotics Research 44 (10-11), p. 1684–1704. Cited by: §I-B. [24] S. Ye, Y. Ge, K. Zheng, et al. (2026) World action models are zero-shot policies. arXiv preprint arXiv:2602.15922. Cited by: §I-B. [25] T. Yuan, Z. Dong, Y. Liu, and H. Zhao (2026) Fast-WAM: do world action models need test-time future imagination?. arXiv preprint arXiv:2603.16666. Cited by: §I-B. [26] A. Huning (1976) Evolutionsstrategie. optimierung technischer systeme nach prinzipien der biologischen evolution. JSTOR. Cited by: §I-C. [27] T. Salimans, J. Ho, X. Chen, S. Sidor, and I. Sutskever (2017) Evolution strategies as a scalable alternative to reinforcement learning. arXiv preprint arXiv:1703.03864. Cited by: §I-C, §I-B, §IV-A. [28] N. Hansen (2016) The CMA evolution strategy: a tutorial. arXiv preprint arXiv:1604.00772. Cited by: §I-C. [29] Y. Li, W. Gong, and S. Li (2023) Multitask evolution strategy with knowledge-guided external sampling. IEEE Transactions on Evolutionary Computation 28 (6), p. 1733–1745. Cited by: §I-C. [30] M. Nomura, Y. Akimoto, and I. Ono (2025) CMA-ES with learning rate adaptation. ACM Transactions on Evolutionary Learning 5 (1), p. 1–28. Cited by: §I-C. [31] L. Sun, B. Xie, Y. Liu, et al. (2026) GeoVLA: empowering 3d representations in vision-language-action models. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Cited by: §V-A. [32] Zhiyuan Robotics (AgiBot) AgiBot g1 general purpose robot. Note: https://w.agibot.com/products/G1Accessed: 2026-7-15 Cited by: §V-A. [33] B. Liu, Y. Zhu, C. Gao, Y. Feng, et al. (2023) LIBERO: benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems 36, p. 44776–44791. Cited by: §V-B. [34] M. J. Kim, K. Pertsch, S. Karamcheti, et al. (2025) OpenVLA: an open-source vision-language-action model. In Proceedings of The 8th Conference on Robot Learning, Vol. 270, p. 2679–2713. Cited by: TABLE I. [35] Y. Zhao, L. Zhao, B. Cheng, G. Yao, X. Wen, and H. Gao (2025) Vla-rail: a real-time asynchronous inference linker for vla models and robots. arXiv preprint arXiv:2512.24673. Cited by: §V-C.