Paper deep dive
VAMPS: Visual and Motor Policies from Sampling-Based Planning
Mohamed Yassine Kabouri, Pietro Noah Crestaz, Quang-Nam Nguyen, Qilong Cheng, Ludovic Righetti, Nicolas Mansard
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Learning robot policies directly on physical systems remains difficult because data collection is costly and policy exploration can be unsafe. We introduce Visual and Motor Policies from Sampling-Based Planning (VAMPS), a framework that uses Model Predictive Path Integral (MPPI) control to train reusable policies without human demonstrations. VAMPS supports two training modes. For one-step proprioceptive policies, it operates iteratively in simulation: the policy warm-starts MPPI, and the refined trajectories provide new supervision as the policy changes. A learned terminal value improves short-horizon planning, while an Implicit Q-Learning (IQL) critic guides the policy update. Iterative refinement outperforms training once on frozen MPPI data, and we transfer the learned locomotion policy to a Unitree Go2. For visuomotor policies, VAMPS operates directly from real-robot data. MPPI uses task-specific state estimates to plan and execute trajectories while recording RGB and sensor observations on a Flexiv Rizon 10S. Action Chunking with Transformers predicts action chunks, reducing the effective prediction horizon, and is trained offline on this fixed dataset. We demonstrate visuomotor pick-and-place and force-aware whiteboard erasing. In the latter task, the policy additionally observes the measured $6$-D wrench and desired normal force. These results show that VAMPS can learn policies either in simulation followed by hardware transfer or directly from autonomously collected real-robot data.
Tags
Links
- Source: https://arxiv.org/abs/2610.05331v1
- Canonical: https://arxiv.org/abs/2610.05331v1
Trouble viewing inline? Open PDF directly →
Full Text
50,919 characters extracted from source content.
Expand or collapse full text
algorithm VAMPS: Visual and Motor Policies from Sampling-Based Planning Mohamed Yassine Kabouri Affiliation: LAAS-CNRS, Université de Toulouse, CNRS, Toulouse, France. Affiliation: Machines in Motion Laboratory, New York University, New York, USA. Pietro Noah Crestaz Affiliation: LAAS-CNRS, Université de Toulouse, CNRS, Toulouse, France. Affiliation: Industrial Engineering Department, University of Trento, Trento, Italy. Quang-Nam Nguyen Affiliation: Machines in Motion Laboratory, New York University, New York, USA. Qilong Cheng Affiliation: Machines in Motion Laboratory, New York University, New York, USA. Ludovic Righetti Affiliation: Machines in Motion Laboratory, New York University, New York, USA. Affiliation: Artificial and Natural Intelligence Toulouse Institute (ANITI), Toulouse. Nicolas Mansard Affiliation: LAAS-CNRS, Université de Toulouse, CNRS, Toulouse, France. Affiliation: Artificial and Natural Intelligence Toulouse Institute (ANITI), Toulouse. Abstract Learning robot policies directly on physical systems remains difficult because data collection is costly and policy exploration can be unsafe. We introduce Visual and Motor Policies from Sampling-Based Planning (VAMPS), a framework that uses Model Predictive Path Integral (MPPI) control to train reusable policies without human demonstrations. VAMPS supports two training modes. For one-step proprioceptive policies, it operates iteratively in simulation: the policy warm-starts MPPI, and the refined trajectories provide new supervision as the policy changes. A learned terminal value improves short-horizon planning, while an Implicit Q-Learning (IQL) critic guides the policy update. Iterative refinement outperforms training once on frozen MPPI data, and we transfer the learned locomotion policy to a Unitree Go2. For visuomotor policies, VAMPS operates directly from real-robot data. MPPI uses task-specific state estimates to plan and execute trajectories while recording RGB and sensor observations on a Flexiv Rizon 10S. Action Chunking with Transformers predicts action chunks, reducing the effective prediction horizon, and is trained offline on this fixed dataset. We demonstrate visuomotor pick-and-place and force-aware whiteboard erasing. In the latter task, the policy additionally observes the measured 66-D wrench and desired normal force. These results show that VAMPS can learn policies either in simulation followed by hardware transfer or directly from autonomously collected real-robot data. †aftertitle: I Introduction Learning directly on physical robots, through interaction and sensor feedback, remains a major challenge, despite being the most direct route to acquiring effective sensorimotor behaviors. This is especially compelling for contact-rich robotics, including pick-and-place manipulation, force-regulated interaction, and quadruped locomotion. Current learning methods do not yet solve this problem at scale: they typically rely on human demonstrations or move exploration to simulation before transferring the resulting policy to hardware. This work takes a step toward methods that can autonomously and efficiently collect sensorimotor experience on robots and turn it into better policies. This objective has been pursued through three main routes. Imitation learning methods such as Action Chunking with Transformers (ACT) [35] and Diffusion Policy [5] clone human teleoperation and learn rich behaviors. Their bottleneck is data: demonstrations are expensive, performance is bounded by the operator, and cloning a fixed dataset leads to compounding errors [27]. DAgger [27] corrects those states by querying an expert policy on states visited by the learned policy. Reinforcement learning [12] needs no demonstration, but it has to explore, and exploration is a bottleneck on contact-rich tasks with short episodes [21]. Trajectory Optimization (TO) [34, 13], in contrast, optimizes individual rollouts for a given cost. While gradient-based TO relies on derivatives that are challenging to compute in contact-rich tasks, sampling-based planners [31] handle non-smooth contact dynamics and run in parallel on GPUs [32]. Yet their search space scales exponentially with the planning horizon [7]. They are commonly used in receding horizon, which requires online replanning and does not directly produce a reusable policy. Fig. 1: Overview of the two VAMPS modes. (a) VAMPS-Iterative: a one-step proprioceptive policy warm-starts MPPI, which refines its behavior and provides new supervision. A terminal value improves short-horizon planning, and an IQL critic weights the policy updates. (b) VAMPS-Batch: MPPI uses task-specific estimates to autonomously collect RGB, sensor and action episodes on the real robot, and ACT trains offline on the resulting fixed dataset. The complementarity between trajectory optimization and policy learning has long been recognized, notably by Guided Policy Search (GPS) [17] and subsequent optimizer-policy coupling methods [23, 19]. This perspective combines the precision of model-based optimization with the fast execution of a learned controller, while avoiding direct exploration by the learned policy. We build on this principle and use MPPI as a behavior generator from a model and a task cost. In this work, we introduce Visual and Motor Policies from Sampling-Based Planning (VAMPS), a framework that distills trajectories generated by MPPI into reusable robot policies without human demonstrations or simulator gradients. MPPI plans with a model, a task cost and task-specific state estimates, while the learned policy maps the observations available at deployment directly to actions. VAMPS supports two training modes that share this planner-to-policy principle. VAMPS-Iterative trains a one-step proprioceptive policy in simulation. Small prediction errors can move such a policy away from the states in a fixed training dataset and compound over time [27]. The policy therefore repeatedly warm-starts MPPI, which refines its behavior and provides new supervision as the policy evolves. This loop does not require simulator gradients. A learned terminal value improves short-horizon planning, while an IQL [15] critic guides the policy update through Advantage-Weighted Regression (AWR) [25]. We train the policy in simulation and transfer it to a real robot. VAMPS-Batch uses MPPI to autonomously collect a real-robot dataset and train a visuomotor policy from it. MPPI uses task-specific state estimates to plan and execute trajectories on the robot while RGB images, robot sensors and actions are recorded. ACT [35] predicts action chunks, reducing the effective prediction horizon. For the tasks considered here, we collect the MPPI episodes once, freeze the dataset and train ACT offline. At deployment, ACT maps images and robot sensors directly to action chunks and does not use the task-specific state estimates. Our contributions are threefold. (1) We introduce Visual and Motor Policies from Sampling-Based Planning (VAMPS), a framework that uses MPPI to train reusable policies either in simulation or directly from real-robot data. (2) VAMPS-Iterative trains one-step proprioceptive policies. We show that iterative refinement outperforms training once on frozen MPPI data, a simple behavioral-cloning coupling between MPPI and the policy is sufficient, and a learned terminal value improves short-horizon planning. We transfer the resulting locomotion policy to a Unitree Go2. (3) VAMPS-Batch uses MPPI to autonomously collect a fixed real-robot dataset without human demonstrations and trains chunked ACT policies on it. We demonstrate this mode on pick-and-place and force-aware whiteboard erasing with a Flexiv Rizon 10S. I Related Work GPS [17] and its visuomotor extension [16] pioneered the idea of using local TO controllers to supervise policy learning. In the same cycle of ideas, [23] formalized the TO-policy coupling via ADMM to enforce agreement between them. [19] later used augmented-Lagrangian updates and sensitivity information to improve this coupling. CACTO-SL instead combines TO with actor-critic learning and uses the policy to warm-start the optimizer [1]. More recently, [11] trained diffusion policies to warm-start gradient-based TO. Their method alternates policy learning and trajectory refinement, and uses the feedback gains returned by the optimizer in the training loss. These methods rely mainly on gradient-based TO and differentiable dynamics. PI2GPS [4] replaces the gradient-based local optimizer with the derivative-free PI2 algorithm. However, it initializes the local policies from kinesthetic demonstrations and pretrains its visual features on a pose-prediction task. This work studies how to scale the GPS idea using modern tools from derivative-free optimization [14], offline RL [18] and imitation learning [35]. VAMPS-Iterative uses GPU-parallel MPPI in receding horizon. It requires a model and a task cost but does not require simulator derivatives, feedback gains or a human warm start. The policy repeatedly warm-starts MPPI, which refines its behavior and provides new supervision. To improve short-horizon planning, we introduce a learned terminal value trained on the planner transitions. To filter the planner actions, we train an IQL critic [15] on MPPI trajectories and policy rollouts, and use its advantage in AWR [25] to give more weight to high-advantage actions. Visuomotor imitation methods such as ACT [35] and Diffusion Policy [5] predict sequences of actions from visual observations. Action-sequence prediction reduces the effective prediction horizon and limits the accumulation of one-step errors. These methods are commonly trained from human demonstrations. VAMPS-Batch instead uses MPPI to collect real-robot RGB, sensor and action episodes, and trains ACT on the resulting fixed dataset. Task-specific state estimates guide MPPI during data collection but are not provided to the deployed policy. For force-aware erasing, MPPI records RGB images, the measured wrench and the desired normal force, and ACT learns a force-conditioned policy from these episodes. We deploy the proprioceptive policy learned by VAMPS-Iterative on a Unitree Go2 and the visuomotor policies learned by VAMPS-Batch on a Flexiv Rizon 10S. I Preliminaries Fig. 2: The hardware that we consider, four frames each in temporal order. Top: bounding policy of the Unitree Go2 under a forward velocity command, with a proprioceptive policy trained in simulation. Middle: pick-and-place on the Flexiv Rizon 10S. Bottom: whiteboard erasing on the same arm. The two arm tasks are visuomotor and are learned directly from hardware data. Videos are provided in the supplementary video. I-A Model Predictive Path Integral (MPPI) MPPI [31] is a sampling-based trajectory optimization method. From the current state 0x_0, it solves the following finite-horizon optimal control problem (OCP): min0:T−1 _u_0:T-1 J(0,)=∑t=0T−1γtc(t,t)+γTcf(T) J(x_0,\,u)= _t=0^T-1γ^tc (x_t,\,u_t )+γ^Tc_f(x_T) (1) s.t. .t. t+1=f(t,t),∀t=0,…,T−1, _t+1=f(x_t,u_t), ∀ t=0,…,T-1, t∈,t∈, _t , _t , Here, =(0,…,T−1)u=(u_0,…,u_T-1) and γ is the discount factor. We use the cost convention c=−rc=-r in the ocp, while the experiments report return, for which higher values are better. MPPI samples K perturbations w(i)∼(0,Σ0:T−1)w^(i) (0, _0:T-1) around the nominal control sequence u. The cost of each perturbed trajectory is J(0,+w(i))J(x_0,u+w^(i)), where the states are obtained by rolling out the dynamics from 0x_0. This cost gives the standard MPPI update: ←+∑i=1Kexp(−1λJ(0,+w(i)))w(i)∑i=1Kexp(−1λJ(0,+w(i))). + _i=1^K (- 1λJ(x_0,u+w^(i)) )\,w^(i) _i=1^K (- 1λJ(x_0,u+w^(i)) ). (2) We run MPPI in a receding-horizon loop. A short horizon of the optimized control sequence is applied to the system, after which the control sequence is shifted and MPPI solves a new OCP from the resulting state. I-B Implicit Q-Learning IQL [15] learns a state value VψV_ψ and an action value QϕQ_φ from a dataset of transitions D. It avoids maximizing QϕQ_φ over actions outside the dataset, which can cause value overestimation [18]. Instead, it approximates the in-sample maximum with the asymmetric squared expectile loss L2τL_2^τ: ℒV(ψ) _V(ψ) =(x,u)∼[L2τ(Qϕ(x,u)−Vψ(x))], =E_(x,u) \! [L_2^τ\! (Q_φ(x,u)-V_ψ(x) ) ], (3) ℒQ(ϕ) _Q(φ) =[(r+γVψ(x′)−Qϕ(x,u))2], =E_D\! [ (r+γ V_ψ(x )-Q_φ(x,u) )^\!2 ], (4) where L2τ(e)=|τ−[e<0]|e2L_2^τ(e)=|τ-1[e<0]|e^2. The resulting advantage is Aϕ(x,u)=Qϕ(x,u)−Vψ(x)A_φ(x,u)=Q_φ(x,u)-V_ψ(x). I-C Action Chunking with Transformers (ACT) ACT [35] is a visuomotor policy that predicts a short action trajectory (called a chunk) of TuT_u consecutive joint-position targets instead of one action. This reduces the accumulation of prediction errors. Let It,cI_t,c denote the RGB image from camera c, tq_t the robot sensor observation, and Ft,c=ResNet(It,c)F_t,c=ResNet(I_t,c) the spatial image features. ACT is trained as a conditional variational autoencoder (CVAE). Its policy uses a Transformer encoder to combine (t,Ft,c,)(q_t,\F_t,c\,z), and a Transformer decoder predicts ^t:t+Tu−1 u_t:t+T_u-1. The latent variable z represents the variation between action trajectories. During training, a separate CVAE encoder infers z from (t,t:t+Tu−1)(q_t,u_t:t+T_u-1). Training minimizes ℒ=∼qϕ[∥t:t+Tu−1−^t:t+Tu−1∥1]+βKLDKL(qϕ(∣t,t:t+Tu−1)∥(,)). splitL=&E_z q_φ [\|u_t:t+T_u-1- u_t:t+T_u-1\|_1 ]\\ &+ _KL\,D_KL\! (q_φ(z _t,u_t:t+T_u-1)\,\|\,N(0,I) ). split (5) At inference the CVAE encoder is discarded and =z=0, the mean of the prior. IV VAMPS: Learning Policies from Sampling-Based Planning VAMPS uses MPPI in two complementary modes. VAMPS-Iterative alternates between MPPI trajectory refinement and policy fitting, whereas VAMPS-Batch first collects a fixed dataset with MPPI and then trains a policy offline. IV-A VAMPS-Iterative Our goal is to learn a policy πθ _θ over a distribution of initial states p0p_0. Rolling out πθ _θ from 0x_0 produces a control sequence whose cost is J in Equation (1); we write Jπθ(0)J _θ(x_0) for that cost. The policy search problem is minθ0∼p0[Jπθ(0)]. _θ\;E_x_0 p_0 [J _θ(x_0) ]. (6) In practice, we sample a finite set of initial states x0(i)i=1N∼p0\x_0^(i)\_i=1^N p_0. Following GPS [22] and [19], we introduce a control sequence (i)u^(i) for each sampled initial state and require these controls to be reproducible by the policy: minθ,(i)i=1N _θ,\,\u^(i)\_i=1^N 1N∑i=1NJ(x0(i),(i)) 1N _i=1^NJ\! (x_0^(i),\,u^(i) ) (7) s.t. .t. t+1(i)=f(xt(i),ut(i)), _t+1^(i)=f (x_t^(i),\,u_t^(i) ), ut(i)=πθ(xt(i)),∀i=1,…,N,∀t. u_t^(i)= _θ\! (x_t^(i) ), ∀\,i=1,…,N,\;∀\,t. We consider three strategies for handling this constraint in Sec. IV-A2. For a Gaussian policy, we express the constraint as a KL divergence between the MPPI trajectory distribution and the policy distribution. IV-A1 Policy-Guided MPPI with a Learned Terminal Value VAMPS-Iterative alternates between trajectory refinement with MPPI and policy fitting. At outer iteration k, we run MPPI in receding-horizon fashion from each sampled initial state x0(i)x_0^(i). At every replanning step, the current policy provides the nominal control sequence by rolling out the dynamics, u¯t(i)=πθ(xt(i)),xt+1(i)=f(xt(i),u¯t(i)), u_t^(i)= _θ(x_t^(i)), x_t+1^(i)=f(x_t^(i), u_t^(i)), (8) and we write ¯(i)=(u¯0(i),…,u¯T−1(i)) u^(i)=( u_0^(i),…, u_T-1^(i)). MPPI refines this sequence, and the refined controls (i)u^(i) supervise the policy. We extend the effective planning horizon through bootstrapping [7] by setting the terminal cost in (1) to cf(xT)=−V^ω(xT)c_f(x_T)=- V_ω(x_T). The minus sign converts the predicted return into a cost. The value function is learned from MPPI transitions by regression onto n-step bootstrapped returns: ℒV^(ω) _ V(ω) =(xt,Gt)∼ℬ[(V^ω(xt)−Gt)2], =E_(x_t,\,G_t) \! [ ( V_ω(x_t)-G_t )^2 ], (9) Gt G_t =∑j=0n−1γjrt+j+γnV^ω−(xt+n), = _j=0^n-1γ^jr_t+j+γ^n V_ω^-(x_t+n), with γ the discount factor, ω−ω^- the parameters of a target network updated by Polyak averaging, and ℬB a dedicated buffer holding only (xt,Gt)(x_t,G_t) pairs from MPPI trajectories. Algorithm 1 VAMPS-Iterative 1: initial-state distribution p0p_0, number of states N, horizon T, outer iterations M, penalty ρ, coupling ∈BC,Penalty,ADMM∈\BC,\,Penalty,\,ADMM\ 2: Sample x0(i)i=1N∼p0\x_0^(i)\_i=1^N p_0 ⊳ once, fixed for all iterations 3: Initialize policy θ, terminal value ω, critic (ψ,ϕ)(ψ,φ) 4: (i)←y^(i)\! 0, ←∅D← , ℬ←∅B← 5: for k=1k=1 to M do 6: for i=1i=1 to N do ⊳ GPU-parallel 7: (x¯t(i),u¯t(i))t=0T−1←rollout πθ ( x^(i)_t,\, u^(i)_t )_t=0^T-1 _θ ⊳ warm start 8: (i)←MPPI(Ji+Φθ((i),(i)))u^(i) (J_i+ _θ(u^(i),y^(i)) ) ⊳ cf=−V^ωc_f=- V_ω 9: k←(xt(i),ut(i))i,tG^k←\(x_t^(i),u_t^(i))\_i,t 10: ℬ←ℬ∪kB ^k 11: ←∪k∪(x¯t(i),u¯t(i))i,tD ^k∪ \( x^(i)_t,\, u^(i)_t) \_i,t 12: Update V^ω V_ω on ℬB ⊳ (9) 13: Update Vψ,QϕV_ψ,\,Q_φ on D by IQL ⊳ (3), (4) 14: Update θ on kG^k ⊳ (17) 15: if ADMM then 16: yt(i)←yt(i)+ut(i)−μθ(xt(i)),∀i,ty_t^(i)← y_t^(i)+u_t^(i)- _θ(x_t^(i)), ∀ i,t ⊳ (18) 17: return θ IV-A2 Planner-Policy Coupling VAMPS-Iterative supports three coupling strategies: BC, Penalty and ADMM. They differ in how they enforce the constraint in (7). We write Ji()=J(x0(i),)J_i(u)=J(x_0^(i),u) and denote the control sequence returned by MPPI as MPPI(Ji)MPPI(J_i). We measure the disagreement between a planner action and the policy with dθ(x,u,y)=‖u+y−πθ(x)‖2,deterministic,DKL(qu+y∥πθ(⋅∣x)),Gaussian,d_θ(x,u,y)= cases\|u+y- _θ(x)\|^2,&deterministic,\\ D_KL\! (q_u+y\,\|\, _θ(· x) ),&Gaussian, cases (10) where qv=(v,σMPPI2)q_v=N(v, _MPPI^2I) is the MPPI trajectory distribution for a Gaussian policy [6]. In ADMM, y=λ/ρ∈ℝnuy=λ/ρ ^n_u is the scaled dual variable associated with the planner-policy consensus constraint, where λ is the corresponding Lagrange multiplier and ρ>0ρ>0 is the penalty parameter. We define the trajectory-level coupling penalty as Φθ((i),(i))=ρ2∑tdθ(xt(i),ut(i),yt(i)). _θ\! (u^(i),y^(i) )= ρ2 _td_θ\! (x_t^(i),u_t^(i),y_t^(i) ). (11) Here, ρ>0ρ>0 is the coupling coefficient. BC coupling. The planner is uncoupled from the current policy and runs MPPI on the task cost alone: (i),k+1←MPPI(Ji).u^(i),k+1 (J_i). (12) Penalty coupling. The coupling Φθ _θ (with (i)=y^(i)=0) is added to the MPPI cost: (i),k+1←MPPI(Ji+Φθ((i),)).u^(i),k+1 \! (J_i+ _θ\! (u^(i),0 ) ). (13) The penalty biases MPPI toward trajectories the current policy can represent. ADMM coupling. The constraint of (7) is the consensus (i)=¯(i)u^(i)= u^(i). Using the scaled dual variables (i)y^(i), the augmented Lagrangian is: ℒρ=∑i=1NJi+∑i=1NΦθ((i),(i)).L_ρ= _i=1^NJ_i+ _i=1^N _θ\! (u^(i),y^(i) ). (14) Its trajectory update is (i),k+1←MPPI(Ji+Φθ((i),(i),k)).u^(i),k+1 \! (J_i+ _θ\! (u^(i),\,y^(i),k ) ). (15) Algorithm 2 VAMPS-Batch 1: task cost J, initial-state distribution p0p_0, episodes E 2: batch←∅D_batch← 3: for i=1i=1 to E do 4: Sample x0(i)∼p0x_0^(i) p_0 5: # Replan online and record data 6: τi←ExecuteOnRobot(MPPI(J(x0(i),))) _i (MPPI(J(x_0^(i),u)) ) 7: batch←batch∪τiD_batch _batch∪\ _i\ 8: Freeze batchD_batch and train ACT with (5) 9: return θ (a) Panda pick-and-place (b) Panda pick-and-place (c) Go2 Fig. 3: The three VAMPS-Iterative coupling strategies. Each task shows a Gaussian policy and a deterministic policy. Shaded bands: 95 % CI over 4 seeds. IV-A3 Critic-Guided Policy Learning At each iteration, we add the refined MPPI trajectories and the policy rollouts used to warm-start MPPI to a replay buffer D. We use both sources for two reasons. First, the warm-start rollouts provide transitions from the current policy, reducing extrapolation when we evaluate Qϕ(x,πθ(x))Q_φ(x, _θ(x)). Second, combining them with the refined MPPI trajectories improves the action coverage used to estimate the advantages. We train VψV_ψ and QϕQ_φ with the IQL losses in (3)-(4). The buffer D is used only for these critic updates. AWR [25] converts the advantage Aϕ(x,u)=Qϕ(x,u)−Vψ(x)A_φ(x,u)=Q_φ(x,u)-V_ψ(x) into the normalized weight wβ(x,u)=exp(Aϕ(x,u)/β)(x′,u′)∼k[exp(Aϕ(x′,u′)/β)],w_β(x,u)= \! (A_φ(x,u)/β )E_(x ,u ) ^k\! [ \! (A_φ(x ,u )/β ) ], (16) where β>0β>0 is the temperature. All three coupling strategies use these weights to update the policy on the current MPPI trajectories k=(xt(i),ut(i),k+1)i,tG^k=\(x_t^(i),u_t^(i),k+1)\_i,t: θk+1=argminθ∑(x,u)∈kwβ(x,u)dθ(x,u,y).θ^k+1= *arg\,min_θ _(x,u) ^kw_β(x,u)\,d_θ(x,u,y). (17) BC and Penalty set y=0y=0 in this update. ADMM uses its current scaled dual variable yky^k. After the policy update, ADMM updates this variable at each time step: yt(i),k+1←yt(i),k+ut(i),k+1−μθk+1(xt(i)),y_t^(i),k+1← y_t^(i),k+u_t^(i),k+1- _θ^k+1\! (x_t^(i) ), (18) where μθ(x)=u∼πθ(⋅∣x)[u] _θ(x)=E_u _θ(· x)[u] for a Gaussian policy and μθ(x)=πθ(x) _θ(x)= _θ(x) for a deterministic policy. The dual variable accumulates the planner-policy residual. IV-B VAMPS-Batch VAMPS-Iterative uses a one-step proprioceptive policy. Small prediction errors can move this policy away from the states in a fixed training dataset and compound over time [27]. Our frozen-data baseline show this effect in Sec. V-F. VAMPS-Iterative addresses it by repeatedly refining the policy trajectories with MPPI and using them as new supervision. VAMPS-Batch instead uses ACT [35], which predicts an action trajectory rather than one action at each step. This reduces the effective decision horizon and motivates training the visuomotor policy once on a fixed dataset. VAMPS-Batch uses MPPI to autonomously collect this dataset on the real robot. MPPI evaluates candidate rollouts in simulation using task-specific state estimates and executes the refined controls on the robot. For example, it uses the cube position for pick-and-place and the contact height z for erasing. During execution, we record synchronized RGB images, robot observations, task sensors and actions. We freeze the resulting episodes and train ACT offline with (5). At deployment, ACT does not use the task-specific state estimates. Its evaluation is reported in Sec. VI. IV-C Algorithm Summary Algorithms 1 and 2 summarize the two VAMPS modes. Both use MPPI to generate supervision from a model and a task cost. VAMPS-Iterative updates the planner, policy and value functions together: the current policy warm-starts MPPI, and each refined trajectory updates the policy before the next outer iteration. VAMPS-Batch instead completes data collection first, freezes the recorded real-robot episodes and then trains ACT offline. V VAMPS-Iterative Experiments & Results V-A Experimental Setup VAMPS tasks. In Panda pick-and-place from MuJoCo Playground [33], a Franka Panda grasps a cube from randomized positions and lifts it to a target. We use this task for the coupling comparison, component ablations and standard baselines. Go2 evaluates the coupling on locomotion and real-robot policy transfer. VAMPS-Iterative architecture. The policy πθ _θ, critic (Vψ,Qϕ)(V_ψ,Q_φ) and terminal value V^ω V_ω use BroNet [24]: a Dense → LayerNorm → ReLU stem and residual blocks of width 512. The critic uses two blocks, while πθ _θ and V^ω V_ω use one. QϕQ_φ is an ensemble of 10 networks [3], whose mean defines the advantage. The policy output layer uses orthogonal initialization with scale 0.010.01. We train all networks with AdamW [20]. VAMPS-Iterative training. At each outer iteration, the critic and terminal value take 15 gradient steps with batch sizes 1024 and 256, respectively; the terminal value uses n=2n=2-step targets. We use IQL expectile τ=0.7τ=0.7, discount γ=0.99γ=0.99, and target update rate τnet=0.005 _net=0.005. The policy takes 10 epochs on Panda and 15 on Go2, with learning rate 5×10−45×10^-4 and AWR temperature β=2.0β=2.0. MPPI uses horizon T=16T=16, DIAL-MPC [32] annealing, and replans every two steps. Panda uses N=80N=80, K=512K=512, Ndiffuse=3N_diffuse=3, and 300 outer iterations. Go2 uses N=50N=50, K=1024K=1024, Ndiffuse=2N_diffuse=2, 50 outer iterations, and domain randomization. VAMPS-Iterative metrics. The primary metric is the mean return evaluated on a test set of 50 initial states per seed; shaded bands indicate 95 % confidence intervals across seeds. Default configuration. Unless stated otherwise, the experiments use VAMPS-Iterative (Algorithm 1) with a Gaussian policy. V-B Q1: How does the coupling strategy between the trajectory optimizer and the policy affect performance? Setup. We run the full method with each of the three coupling strategies of Sec. IV-A2, and we do so twice: once with a Gaussian policy and once with a deterministic one. This gives six conditions per task, and we run each of them with four seeds. We denote the configurations VAMPS-Iterative (BC), VAMPS-Iterative (Penalty) and VAMPS-Iterative (ADMM). BC uses ρ=0ρ=0, Penalty uses ρ=0.005ρ=0.005 with the dual variables frozen, and ADMM uses ρ=0.005ρ=0.005 with the dual variables active. Result. On the Panda task, BC and Penalty pass a success rate of 0.900.90 within about 3030 iterations and then hold a plateau just under 1.01.0, with both the Gaussian and the deterministic policy (Fig. 3). ADMM climbs as fast at first but settles lower, at 0.880.88 with the Gaussian policy and 0.900.90 with the deterministic one. The return shows the same: BC and Penalty plateau between 10101010 and 10601060, with Penalty above BC by about 4040 with both policies. ADMM plateaus between 860860 and 875875. The Go2 task gives the same result. BC and Penalty again behave identically. ADMM is again the worst, and its instability is more visible here. The Go2 curves start from a high return because of the orthogonal action head initialized at a scale of 0.010.01: the untrained network outputs a near-zero action, which under position control is the default joint configuration. Analysis. We attribute the instability of the ADMM variant to the dual variables y. In the policy step (17), they shift the regression target from u to +u+y. The right panel of Fig. 4 shows, at each outer iteration and horizon step t, the 95th percentile of |yt,j(i)||y_t,j^(i)| over the N initial states and the action coordinates. A cell past 11 means at least 5%5\% of those entries put +u+y outside [−1,1][-1,1], the range of πθ _θ. Fig. 4: Left: median primal residual over four seeds: BC already follows the planner to 0.240.24 with no consensus enforced. Right: 95th percentile of |yt,j(i)||y_t,j^(i)| over states and action coordinates, on one Go2 seed. Colorbar in action units; 11 is the bound of πθ _θ. With y set to zero, Penalty at ρ=0.005ρ=0.005 brings no gain over BC, and it brings no loss either. This is consistent with Sec. V-E, where performance degrades as ρ grows. We hypothesize the size of the policy explains this, and the left panel of Fig. 4 supports it. The coupling term exists to keep the planner inside the set of trajectories that the policy is able to reproduce, which is a real constraint when the policy is represented by a low-capacity neural network, as in GPS [17]. Our policy reproduces the MPPI controls with no coupling. The primal residual falls to 0.240.24 within 3030 iterations and stays there, and the penalty only brings it to 0.210.21. The consensus constraint is therefore almost satisfied. The dual updates move it in the opposite direction, to 0.260.26. The displaced target of Fig. 4 pulls the planner and the policy apart rather than together, which is the behavior the method is meant to prevent. We keep Penalty as the proposed VAMPS-Iterative coupling and use it in the remaining experiments. V-C Q2: Does the learned terminal value function compensate for a shorter horizon? Setup. We evaluate the terminal value V^ω V_ω trained at each iteration on the MPPI transitions accumulated in ℬB. It accounts for the part of the task that lies beyond the MPPI horizon, so we vary the horizon as well: T=16T=16, T=8T=8 and T=4T=4, each with and without V^ω V_ω, for four seeds each. Fig. 5: Test return on Panda pick-and-place. Left: planning horizons T∈16,8,4T∈\16,8,4\ with and without the learned terminal value (TV). Right: VAMPS-Iterative without a critic and with the critic for different AWR temperatures β; both retain the terminal value. Shaded bands: 95 % CI across seeds. Result. As expected, the terminal value improves the final policy return at every horizon, and the gain grows as the horizon shrinks (Fig. 5, left). It is large at T=4T=4, still clear at T=8T=8, and small at T=16T=16. The terminal value is not a replacement for a long horizon, because T=4T=4 with it stays well below T=8T=8 without it. The figure therefore shows that a reasonable horizon is important, and that the terminal value lets us reduce the computational budget of MPPI by shortening the rollout. V-D Q3: Does the offline RL critic improve performance? Setup. We compare VAMPS-Iterative with a no-critic ablation, which sets the weights in (17) to one and therefore fits the policy by uniform behavioral cloning on Panda pick-and-place. Both use a Gaussian policy and retain the learned terminal value. We use four seeds per condition. We sweep the AWR temperature β over the values in Fig. 5. Result. VAMPS improves slightly over VAMPS-no-critic for the tested temperatures from β=2β=2 to β=50β=50. At β=1β=1 the final return is similar to VAMPS-no-critic, while β=0.3β=0.3 gives a lower return. The offline RL critic therefore brings a small gain on Panda pick-and-place for a suitable temperature. V-E Q4: How sensitive is performance to the penalty coefficient and batch size? Setup. We vary the penalty coefficient ρ and the batch size, one at a time, over the values shown in the legend of Fig. 6. All other hyperparameters keep their values from Sec. V-A. All runs use a Gaussian policy. Fig. 6: Sensitivity to the hyperparameters. Left: penalty coefficient ρ, with the batch size fixed at 10241024. Right: batch size, with ρ fixed at 0.0050.005. Shaded bands: 95 % CI over 4 seeds. Result. The batch size has no effect, since the five curves overlap over the whole run (Fig. 6). The penalty coefficient behaves as a threshold rather than as a gradual trade-off. Values of ρ=0.005ρ=0.005 and ρ=0.01ρ=0.01 give the same return, while ρ≥0.1ρ≥ 0.1 drops to a much lower level within the first iterations and stays there. The policy is untrained at the beginning, so a large ρ makes the penalty dominate the MPPI cost. V-F Q5: How do standard baselines perform in this regime? Setup. We compare VAMPS-Iterative against PPO [28], frozen BC and frozen IQL on the Panda pick-and-place task, with four seeds and 5050 test initial states. PPO learns from the same reward. We train it at three episode lengths: 8080, which matches VAMPS-Iterative, and 500500 and 10001000. Evaluation is always 8080 steps. BC and IQL learn offline from one pass of MPPI transitions. We give PPO the same simulator budget as VAMPS-Iterative, ×10101.6\!×\!10^10 steps. Of that budget VAMPS-Iterative uses only ×1061.3\!×\!10^6 policy–environment interaction steps, in episodes of length 8080, the rest are planner rollouts, and those policy steps go into the critic’s replay buffer. Every PPO step is a policy-environment interaction step. TABLE I: Baselines on Panda: mean ± 95 % CI over four seeds at each method’s best checkpoint. PPO is trained at the listed episode length and evaluated over 8080 steps, as VAMPS-Iterative. Method Episode length Return Success PPO 8080 279±84279± 84 0.000.00 PPO 500500 975±10975± 10 0.98±0.030.98 0.03 PPO 10001000 894±163894± 163 0.74±0.490.74± 0.49 BC (frozen) – 290±85290± 85 0.02±0.030.02± 0.03 IQL (frozen) – 240±40240± 40 0.01±0.010.01± 0.01 VAMPS-Iterative 8080 1109±761109 76 0.98±0.020.98 0.02 Result. Table I shows that PPO trained at episode length 8080 stays at 279279 return and 0.000.00 success where the gripper moves toward the cube and never picks it up. Increasing the training episode to 500500 raises the mean to 975975 / 0.980.98; at 10001000 the mean is 894894 / 0.740.74, pulled down by one seed that also never grasps, and stays below VAMPS-Iterative. Frozen BC and IQL are trained on one pass of successful MPPI trajectories, yet they stay with PPO at episode length 8080 (290290 / 0.020.02 and 240240 / 0.010.01). VAMPS-Iterative, at episode length 8080, reaches 11091109 / 0.980.98. It passes a return of 10001000 after ×1093.6\!×\!10^9 simulator steps, of which only ×1052.9\!×\!10^5 are policy-environment interaction steps used to warm-start the planner. The key difference is exploration. At 5050 Hz, VAMPS-Iterative uses 8080-step (1.61.6 s) episodes, while PPO needs 500500-step (1010 s) episodes of an untrained policy to solve the task, which is unsafe on hardware. VAMPS-Iterative instead explores with MPPI on a model. At every iteration, the current policy warm-starts MPPI, which refines its actions and returns new trajectories to update the policy. This feedback loop focuses planning around the current policy and provides new supervision as the policy changes. Frozen BC compounds errors outside the MPPI states [27], while frozen IQL has no coverage beyond its single pass of 6,3206,320 transitions [15, 18]. V-G Q6: Does the policy transfer to real hardware? We deploy the proprioceptive policy trained in simulation on a physical Unitree Go2 robot. Fig. 2 shows four frames of one bounding cycle recorded during such a run, where the robot moves under the commanded velocity from the joystick. We used domain randomization [30]. Each of the N initial states carries its own randomized model, drawn once before training and then held fixed for the whole run. We randomize the friction of the floor, the mass of every link, a payload of up to one kilogram on the torso, the position of the torso center of mass, the joint friction and armature, and the PD gains of the position actuators. MPPI plans in the perturbed model of its own scenario, so the policy is fitted on solutions that already account for the model mismatch. VI VAMPS-Batch Experiments & Results Both VAMPS-Batch experiments run on a 7-DOF Flexiv Rizon 10S with a gripper and a side-mounted Orbbec Femto Bolt camera. MPPI evaluates 20482048 sampled rollouts in simulation over a horizon T=20T=20. It replans in receding horizon and executes the refined controls on the robot at 2525 Hz through an impedance controller. We record RGB images, sensor observations and actions, and train ACT on these real-robot episodes using the HuggingFace LeRobot implementation [2, 35]. At inference, the full action trajectory is executed open-loop before ACT is queried again. VI-A Experiment 1: Visuomotor pick-and-place Setup. The robot grasps a cube from a randomized position on the table and drops it in a fixed bowl. We localize the cube with HSV color filtering and a calibrated homography. This estimate is used only by MPPI during data collection. We autonomously collect 100 MPPI episodes. The policy predicts a chunk of Tu=100T_u=100 joint-position targets. Result. The VAMPS-Batch policy succeeds in 8 of 10 trials from randomized cube positions. The middle row of Fig. 2 shows four frames of one successful execution. Fig. 7: Desired and measured normal force during one successful VAMPS-Batch erasing trial shown in Fig. 2. VI-B Experiment 2: Visuomotor and force-aware erasing policy Setup. The robot erases a whiteboard placed on the table. The eraser has to stay in contact with the board while the arm moves over it, so MPPI needs the height at which to hold the eraser: too high and it loses contact and does not erase, too low and it presses through the board. During data collection, we obtain this height with the task-space impedance-admittance nonlinear MPC of [29]. It regulates the contact and gives the z position MPPI plans around. This contact-height estimate is used only during data collection. We collect demonstrations at discrete desired normal-force setpoints from 55 to 1414 N to cover different contact conditions. The resulting dataset contains 84 episodes (1.591.59 hours on the robot). For this task, ACT also takes a second camera mounted on the wrist. Its observation at time t contains the two synchronized images and a 1414-D vector with the seven joint positions, the measured 66-D wrench and the desired normal force. ACT reads the wrench at t without averaging a force window. The policy predicts a chunk of Tu=50T_u=50 joint-position targets (22 s at 2525 Hz), executed open-loop as in pick-and-place. Result. The same VAMPS-Batch checkpoint succeeds in four consecutive trials and erases the writing in all of them. Fig. 7 shows the desired and measured normal force during one successful trial. The bottom row of Fig. 2 shows four frames of the same execution. These preliminary results show that VAMPS-Batch can train visuomotor policies from autonomously collected real-robot data without human demonstrations. We plan to extend this mode to more sensor-based tasks in future work. VII Conclusions We presented VAMPS, a framework that uses a sampling-based planner to train reusable robot policies without human demonstrations or a differentiable simulator. We studied two modes of this framework. VAMPS-Iterative alternates MPPI trajectory refinement and policy learning for one-step proprioceptive policies. Our experiments show that iterative refinement outperforms training once on frozen MPPI data, simple planner-policy coupling is sufficient, and a learned terminal value improves short-horizon planning. We transfer the resulting locomotion policy to a Unitree Go2. VAMPS-Batch uses MPPI to autonomously collect a fixed real-robot dataset and trains chunked visuomotor policies offline with ACT. The dataset contains RGB images, robot observations, task sensors and actions. We demonstrate this mode on pick-and-place and force-aware whiteboard erasing with a Flexiv Rizon 10S. The VAMPS-Batch hardware results are preliminary, and we plan to extend it to more sensor-based tasks. For long-horizon tasks, TAMP [9] could provide a high-level planner. Scene generators such as SimFoundry [26] could build simulations from real footage, allowing VAMPS to collect data and learn visuomotor policies in a real-to-sim-to-real loop. Future work will also study tighter planner–policy feedback in architectures such as TD-MPC2 [10] and CAST [8]. References [1] E. Alboni, G. Grandesso, G. P. Rosati Papini, J. Carpentier, and A. Del Prete (2024) CACTO-SL: Using Sobolev learning to improve continuous actor-critic with trajectory optimization. In Proceedings of the Learning for Dynamics & Control Conference (L4DC), Proceedings of Machine Learning Research, Vol. 242. External Links: Link Cited by: §I. [2] R. Cadene, S. Alibert, A. Soare, Q. Gallouedec, A. Zouitine, S. Palma, P. Kooijmans, M. Aractingi, M. Shukor, D. Aubakirova, M. Russi, F. Capuano, C. Pascal, J. Choghari, K. Meftah, M. Ellerbach, J. Moss, and T. Wolf (2024) LeRobot: state-of-the-art machine learning for real-world robotics in pytorch. Note: https://github.com/huggingface/lerobot Cited by: §VI. [3] W. Chang, M. Henaff, B. Amos, G. Dudek, and S. Fujimoto (2026) The surprising difficulty of search in model-based reinforcement learning. Note: arXiv:2601.21306 External Links: Link Cited by: §V-A. [4] Y. Chebotar, M. Kalakrishnan, A. Yahya, A. Li, S. Schaal, and S. Levine (2017) Path integral guided policy search. IEEE International Conference on Robotics and Automation (ICRA). External Links: Link Cited by: §I. [5] C. Chi, Z. Xu, S. Feng, E. Cousineau, Y. Du, B. Burchfiel, R. Tedrake, and S. Song (2024) Diffusion policy: visuomotor policy learning via action diffusion. The International Journal of Robotics Research (IJRR). Cited by: §I, §I. [6] T. M. Cover and J. A. Thomas (2006) Elements of information theory (wiley series in telecommunications and signal processing). Wiley-Interscience. External Links: ISBN 0471241954 Cited by: §IV-A2. [7] P. N. Crestaz, L. De Matteis, E. Chane-Sane, N. Mansard, and A. Del Prete (2026) TD-CD-MPPI: temporal-difference constraint-discounted model predictive path integral control. IEEE Robotics and Automation Letters 11 (1). External Links: Document Cited by: §I, §IV-A1. [8] P. N. Crestaz, M. Y. Kabouri, N. Mansard, and A. D. Prete (2026) CAST: alternating state-value targets and expanded policy gradients for model-based reinforcement learning. Note: arXiv:2609.08853 External Links: Link Cited by: §VII. [9] C. R. Garrett, R. Chitnis, R. Holladay, B. Kim, T. Silver, L. P. Kaelbling, and T. Lozano-Pérez (2020) Integrated task and motion planning. Note: arXiv:2010.01083 External Links: Link Cited by: §VII. [10] N. Hansen, H. Su, and X. Wang (2024) TD-MPC2: scalable, robust world models for continuous control. Note: arXiv:2310.16828 External Links: Link Cited by: §VII. [11] T. L. Hellard, F. N. Tiofack, Q. L. Lidec, and J. Carpentier (2026) Accelerating trajectory optimization with sobolev-trained diffusion policies. External Links: Link Cited by: §I. [12] J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter (2019) Learning agile and dynamic motor skills for legged robots. Science Robotics 4. External Links: Link Cited by: §I. [13] A. Jordana, S. Kleff, A. Meduri, J. Carpentier, N. Mansard, and L. Righetti (2025) Structure-exploiting sequential quadratic programming for model-predictive control. IEEE Transactions on Robotics 41 (). External Links: Document Cited by: §I. [14] A. Jordana, J. Zhang, J. Amigo, and L. Righetti (2025) An introduction to zero-order optimization techniques for robotics. Note: arXiv:2506.22087 External Links: Link Cited by: §I. [15] I. Kostrikov, A. Nair, and S. Levine (2022) Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations (ICLR), External Links: Link Cited by: §I, §I, §I-B, §V-F. [16] S. Levine, C. Finn, T. Darrell, and P. Abbeel (2016) End-to-end training of deep visuomotor policies. Journal of Machine Learning Research (JMLR) 17 (1). External Links: ISSN 1532-4435 Cited by: §I. [17] S. Levine and V. Koltun (2013) Guided policy search. In Proceedings of the International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research, Vol. 28. External Links: Link Cited by: §I, §I, §V-B. [18] S. Levine, A. Kumar, G. Tucker, and J. Fu (2020) Offline reinforcement learning: tutorial, review, and perspectives on open problems. Note: arXiv:2005.01643 External Links: Link Cited by: §I, §I-B, §V-F. [19] Q. L. Lidec, W. Jallet, I. Laptev, C. Schmid, and J. Carpentier (2023) Enforcing the consensus between trajectory optimization and policy learning for precise robot control. In IEEE International Conference on Robotics and Automation (ICRA), Vol. . External Links: Document Cited by: §I, §I, §IV-A. [20] I. Loshchilov and F. Hutter (2019) Decoupled weight decay regularization. In International Conference on Learning Representations, External Links: Link Cited by: §V-A. [21] X. Mao, Y. XU, Z. Sun, E. Miller, D. Layeghi, and M. Mistry (2025) Learning long-horizon robot manipulation skills via privileged action. In Conference on Robot Learning (CoRL), External Links: Link Cited by: §I. [22] W. H. Montgomery and S. Levine (2016) Guided policy search via approximate mirror descent. In Advances in Neural Information Processing Systems, Vol. 29, p. . External Links: Link Cited by: §IV-A. [23] I. Mordatch and E. Todorov (2014) Combining the benefits of function approximation and trajectory optimization. In Robotics: Science and Systems, External Links: Link Cited by: §I, §I. [24] M. Nauman, M. Ostaszewski, K. Jankowski, P. Miłoś, and M. Cygan (2024) Bigger, regularized, optimistic: scaling for compute and sample efficient continuous control. In Conference on Neural Information Processing Systems, External Links: Link Cited by: §V-A. [25] X. B. Peng, A. Kumar, G. Zhang, and S. Levine (2019) Advantage-weighted regression: simple and scalable off-policy reinforcement learning. Note: arXiv:1910.00177 External Links: Link Cited by: §I, §I, §IV-A3. [26] N. Ranawaka, J. Wong, W. Pai, W. Chu, T. Dai, M. Moghani, H. Yin, Y. Jiang, W. Durbano, B. Huynh, Y. Fang, D. Xu, R. Zhang, L. Fei-Fei, L. Fan, B. Wen, A. Mandlekar, and Y. Zhu (2026) SimFoundry: modular and automated scene generation for policy learning and evaluation. Note: arXiv:2606.28276 External Links: Link Cited by: §VII. [27] S. Ross, G. Gordon, and D. Bagnell (2011) A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), Vol. 15. External Links: Link Cited by: §I, §I, §IV-B, §V-F. [28] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. Note: arXiv:1707.06347 External Links: Link Cited by: §V-F. [29] (2026) Task-space motion-force nonlinear model predictive control for precise and compliant control under uncertainties. Note: Under review Cited by: §VI-B. [30] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel (2017) Domain randomization for transferring deep neural networks from simulation to the real world. IEEE International Conference on Intelligent Robots and Systems. External Links: Link Cited by: §V-G. [31] G. Williams, P. Drews, B. Goldfain, J. M. Rehg, and E. A. Theodorou (2017) Information-theoretic model predictive control: theory and applications to autonomous driving. IEEE Transactions on Robotics 34, p. 1603–1622. External Links: Link Cited by: §I, §I-A. [32] H. Xue, C. Pan, Z. Yi, G. Qu, and G. Shi (2024) Full-order sampling-based mpc for torque-level locomotion control via diffusion-style annealing. IEEE International Conference on Robotics and Automation (ICRA). External Links: Link Cited by: §I, §V-A. [33] K. Zakka, B. Tabanpour, Q. Liao, M. Haiderbhai, S. Holt, J. Y. Luo, A. Allshire, E. Frey, K. Sreenath, L. A. Kahrs, C. Sferrazza, Y. Tassa, and P. Abbeel (2025) MuJoCo playground. External Links: Link Cited by: §V-A. [34] H. Zhao, L. Righetti, and M. Khadiv (2026) Hippo: high-performance interior-point and projection-based solver for generic constrained trajectory optimization. IEEE Robotics and Automation Letters 11. External Links: Document Cited by: §I. [35] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn (2023) Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware. In Proceedings of Robotics: Science and Systems, External Links: Document Cited by: §I, §I, §I, §I, §I-C, §IV-B, §VI.