Paper deep dive
RAPID: Robot Agentic Programming from Demonstrations
Yuyao Liu, Jiayuan Mao, David Hsu, Leslie Pack Kaelbling, TomƔs Lozano-PƩrez
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 9/26/2026, 3:08:41 AM
Summary
The paper introduces RAPID (Robot Agentic Programming from Demonstrations), a framework that automatically generates, verifies, and refines robot manipulation programs from a single visual human demonstration. RAPID utilizes an iterative agentic coding loop to infer task specifications, action primitives, and verification environments. It employs an object-centric relational program representation where primitives are trajectory-optimization programs and strategies are composed via relational constraints, enabling generalization across varying object poses, shapes, materials, and environments. The system was evaluated in simulation on nonprehensile and prehensile manipulation tasks and successfully deployed on a real Franka robot arm.
Entities (7)
Relation Signals (6)
RAPID ā generatesfrom ā Visual Human Demonstration
confidence 97% Ā· automatically generates, verifies, and refines robot programs, given a single visual human demonstration
RAPID ā deployson ā Franka
confidence 95% Ā· We also successfully deployed it on a real Franka arm
RAPID ā uses ā Object-Centric Relational Program
confidence 95% Ā· RAPID uses an object-centric relational program representation that focuses on the underlying structure of the demonstrated strategy
Object-Centric Relational Program ā enables ā Generalization
confidence 93% Ā· To make the resulting program reusable beyond the demonstration setting... generalization over object pose, shape, material, and environment
RAPID ā evaluatedon ā LIBERO-PRO
confidence 92% Ā· We also successfully deployed it on a real Franka arm and evaluated on all eight nonprehensile tasks... LIBERO-Pro benchmark
Coding Agent ā partof ā RAPID
confidence 90% Ā· RAPID... iterative agentic loop... coding agent generates a primitive
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Coding agents have demonstrated enormous success in solving complex programming problems. To leverage their potential for robot systems, this work introduces Robot Agentic Programming from Demonstrations (RAPID), which automatically generates, verifies, and refines robot programs, given a single visual human demonstration. The iterative agentic loop of code refinement requires several key ingredients: (i) a testable task specification, (ii) action primitives for robot execution, and (iii) an interactive environment for program execution and verification. RAPID infers all three from the demonstration automatically. To make the resulting program reusable beyond the demonstration setting, RAPID uses an object-centric relational program representation that focuses on the underlying structure of the demonstrated strategy rather than the specific motion per se: it expresses the action primitives as trajectory-optimization programs that realize object-level motion effects, while composing them through relational constraints that capture scene-specific geometry at run time. We evaluated RAPID in simulation on eight challenging contact-rich nonprehensile manipulation tasks as well as general prehensile manipulation tasks in the LIBERO-Pro benchmark. We also successfully deployed it on a real Franka arm and evaluated on all eight nonprehensile tasks. In all experiments, RAPID demonstrated strong performance, with generalization over object pose, shape, material, and environment. Website: this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2609.30249v1
- Canonical: https://arxiv.org/abs/2609.30249v1
Trouble viewing inline? Open PDF directly ā
Full Text
52,523 characters extracted from source content.
Expand or collapse full text
RAPID: Robot Agentic Programming from Demonstrations Yuyao Liu Affiliation: Massachusetts Institute of Technology National University of Singapore Jiayuan Mao David Hsu Affiliation: Massachusetts Institute of Technology National University of Singapore Affiliation: University of Pennsylvania NVIDIA Leslie Pack Kaelbling TomĆ”s Lozano-PĆ©rez ā thanks: ā Equally advising.ā thanks: Corresponding author: Yuyao Liu (yuyaoliu@mit.edu).ā thanks: Work done while Yuyao Liu was visiting NUS. Abstract Coding agents have demonstrated enormous success in solving complex programming problems. To leverage their potential for robot systems, this work introduces Robot Agentic Programming from Demonstrations (RAPID), which automatically generates, verifies, and refines robot programs, given a single visual human demonstration. The iterative agentic loop of code refinement requires several key ingredients: (i) a testable task specification, (i) action primitives for robot execution, and (i) an interactive environment for program execution and verification. RAPID infers all three from the demonstration automatically. To make the resulting program reusable beyond the demonstration setting, RAPID uses an object-centric relational program representation that focuses on the underlying structure of the demonstrated strategy rather than the specific motion per se: it expresses the action primitives as trajectory-optimization programs that realize object-level motion effects, while composing them through relational constraints that capture scene-specific geometry at run time. We evaluated RAPID in simulation on eight challenging contact-rich nonprehensile manipulation tasks as well as general prehensile manipulation tasks in the LIBERO-Pro benchmark. We also successfully deployed it on a real Franka arm and evaluated on all eight nonprehensile tasks. In all experiments, RAPID demonstrated strong performance, with generalization over object pose, shape, material, and environment. Website: https://yuyaoliu.me/projects/rapid. I Introduction Recently, coding agents [1, 2] have shown remarkable capabilities in developing complex software systems, sparking growing interest in solving robotic manipulation tasks through programming [3, 4, 5, 6, 7, 8, 9]. Consider how a human engineer would develop a robot program for the task in Fig. 1 before the advent of coding agents: given a task (pick up the cereal box), they would implement a program representing the solution strategy (press moveā ā ā ), and verify the program on test cases and iteratively debug it. This workflow mirrors the core strength of coding agents: not one-shot code generation, but the iterative cycle of generating, executing, verifying, and revising a program, commonly referred to as the agentic coding loop [10, 11]. Fig. 1: RAPID (Robot Agentic Programming from Demonstrations) generates a reusable manipulation program from a single visual human demonstration; the program generalizes across variations in object appearance, pose, geometry, physical properties, and scene configuration. However, bringing this agentic coding loop to robotic manipulation requires several ingredients that are not readily available: (a) a testable task specification that defines the intended outcome, (b) manipulation primitives that bridge program-level reasoning and low-level execution, and (c) interactive environments for verification. Existing robotic coding-agent systems typically assume these ingredients are provided a priori, through ground-truth rewards and success signals, human-engineered primitives, and resettable environments. This raises a fundamental question: how can we close the agentic coding loop without assuming that these ingredients are provided a priori? Our key idea is to use demonstrations as the programming interface for supplying the task- and motion-level information needed by the agentic coding loop. A demonstration directly shows a feasible way to accomplish a task and, together with a language description, conveys the intended outcome. Motivated by this perspective, we introduce Robot Agentic Programming from Demonstrations (RAPID), a framework for generating and iteratively refining robot programs from a single visual human demonstration with a coding agent (Fig. 2). From the demonstration, RAPID infers the task specification, constructs manipulation primitives and composes them into a strategy. It further reconstructs an interactive simulation environment from the demonstration for program verification and refinement, and generates scene variants to improve generalization. In this way, a single demonstration bootstraps the task specification, manipulation primitives, and verification environment for closing the agentic coding loop. Fig. 2: Overview of RAPID. E is a simulation environment reconstructed from the demonstration, PiP_iās are manipulation primitives, cic_iās are relational constraints connecting two consecutive primitives, and Ī is the strategy program generated by the coding agent. Rather than merely reproducing the demonstration, we aim to extract a reusable and generalizable manipulation program. Programs that directly encode poses, distances, or waypoints may succeed in the demonstrated scene, but can fail when the scene changes. In this work, we propose an object-centric relational program representation, where the solution strategy is represented by object-centric relations and effects. Specifically, RAPID writes primitives as local trajectory-optimization programs that realize object-level motion effects, such as pushing an object by a displacement or flipping it by an angle, while resolving scene-dependent geometry at execution time. At the task level, primitives are composed into a strategy through relational constraints that determine how the outcome of one phase specifies the next. For example, in Fig. 3, rather than replaying a fixed motion, the program first computes the goal position from the relation between the brown obstacle and the white fixture, and then determines how far to move the target object during the current primitive (press move) based on the state produced by the preceding primitive (flip). By expressing primitives and their composition relationally, RAPID preserves the invariant structure of the demonstrated strategy while adapting to changes in object appearance, pose, geometry, physical properties, and scene configuration. The remaining challenge is verification: the coding agent needs an interactive environment in which candidate programs can be executed, evaluated, and revised. RAPID reconstructs such an environment in simulation from the demonstration and augments it with scene variants generated by perturbing the demonstrated scene. The coding agent then generates task-specific filters that discard variants that are physically infeasible or inconsistent with the task. Afterwards, the coding agent executes candidate programs across this verification set and iteratively revises them based on observed failures. In this work, we focus on nonprehensile manipulation, a particularly challenging domain in robotic manipulation. Many prehensile tasks can be composed from a compact and well-established set of reusable primitives [12], such as pick, move in free space, and place. Nonprehensile manipulation, by contrast, involves a much broader range of geometry- and contact-dependent interactions, including pushing [13], pivoting [14], toppling [15], flipping [16], and sliding [17]. Capturing this diversity with a fixed primitive library is difficult, making manual primitive design a substantial engineering burden. Nonprehensile manipulation therefore provides a natural testbed for studying whether a coding agent can construct reusable manipulation primitives from a single demonstration rather than relying on human engineering. We evaluate RAPID on eight contact-rich, multi-step nonprehensile manipulation tasks in simulation, asking whether programs constructed from a single demonstrated instance can generalize to substantial variations in object appearance, pose, geometry, physical properties, and scene configuration. RAPID outperforms baselines without relying on pre-specified success signals, human-engineered manipulation primitives or provided resettable environments. To evaluate the generality of RAPID beyond nonprehensile manipulation, we additionally test it on LIBERO-Pro [18], a simulation benchmark featuring prehensile manipulation tasks. Finally, we demonstrate that RAPID can be deployed effectively on a real robotic system. I Related Work Programming for Robotics. Code as Policies [9] uses robot programs to connect language reasoning with control. Related work generates programs [19, 20], geometric constraints [21, 22], rewards [23, 24], and simulation environments [25]. Agentic programming adds iterative execution and refinement: CaP-X [6] and related systems [4, 8, 3, 26] improve programs through feedback, while Voyager [27] and skill-discovery approaches [5, 28] grow reusable libraries. However, these approaches typically rely on task success signals, human-engineered primitives, or verification environments supplied a priori. In contrast, RAPID derives these ingredients from a single visual human demonstration to close the agentic coding loop. Learning from Few Demonstrations. Learning from few demonstrations benefits from representations that preserve task-relevant structure across objects and scenes. Spatial abstractions support transfer through visual alignment [29], object geometry and correspondence [30, 31, 32, 33], and contact or interaction structure [34, 35, 36, 37]. Demonstrations can also be decomposed into subgoals and primitives [38, 39]. Structured skill representations enable long-horizon planning and composition through contact retargeting, symbolic models, or geometric reasoning [40, 41, 42, 43, 44]. However, these representations are generally embedded in fixed learning or planning pipelines. In contrast, RAPID combines an object-centric relational program representation of manipulation effects and composition constraints with agentic programming to iteratively construct, verify, and refine transferable manipulation strategies. Nonprehensile Manipulation. Nonprehensile manipulation exploits environmental contact to reposition objects and enable otherwise difficult grasps [45]. Prior work develops mechanics-based models [46, 14], contact-aware planners [47, 15, 17], and optimization methods for trajectories and primitives [48, 49, 50, 51]. Learning-based approaches acquire contact-rich policies [52, 53, 54] or compose predefined primitives [55]. However, these methods typically rely on explicit task specifications or task-specific policy training. In contrast, RAPID approaches nonprehensile manipulation through agentic programming from a single visual human demonstration, automatically constructing and iteratively refining the required primitives and their composition. I Overview of RAPID Problem Statement. We consider programming a reusable strategy for quasi-static rigid-body manipulation from a single visual human demonstration D and a language description l. The program should solve a family of task instances ā°E, each comprising a scene-instruction pair (E,l)(E,l), with similar objectives but varying object appearance, geometry, pose, physical properties, and scene configuration. No manipulation primitives, explicit success or reward signals, or instantiated simulation environments are provided a priori. Given only (D,l)(D,l), the agent aims to construct executable primitives =P1,ā¦,PNP=\P_1,ā¦,P_N\, which produce low-level control signals, and compose them into a strategy Ī that captures the demonstrationās underlying structure and generalizes to unseen task instances (Eā,lā)āā°(E ,l ) : (D,l)ā¦(,Ī ),Ī ā”(Eā,lā,)ā succeeds for ā(Eā,lā)āā°.(D,l) (P, ), (E ,l ;P) succeeds for (E ,l ) . Program Representation. We introduce an object-centric relational program representation that enables strategies constructed by the agent to generalize across different scenes. RAPID represents each primitive as a geometry- and physics-aware local trajectory-optimization program that achieves an object-level effect. A strategy then composes these primitives through relational constraints that specify the desired effect of each primitive. We detail this representation in Sec. IV. Strategy Construction. As shown in Fig. 2, RAPID closes the agentic coding loop by deriving task specifications, manipulation primitives, and verification environments from (D,l)(D,l). It segments the demonstration into several phases, infers a subgoal and success predicate for each phase, and performs real-to-sim reconstruction to obtain interactive simulation environments. RAPID further synthesizes feasible scene variants for additional verification. For each phase, the coding agent generates a primitive and iteratively executes, verifies, and revises it across the reconstructed environment and its variants. RAPID then infers the overall task goal and success predicate, composes the primitives into a strategy, and refines the composition through the same loop. We describe this construction and verification process in Sec. V. Strategy Deployment. At deployment, given an RGB-D observation of a novel scene and a language instruction, RAPID reconstructs the scene in simulation, instantiates the strategy in the reconstructed scene, and executes the resulting trajectory on the robot. Lightweight semantic binding assigns scene objects to the semantic roles in the strategy, while the strategy and primitives themselves remain unchanged. IV Object-Centric Relational Programs We now define our object-centric relational program representation: primitives (Sec. IV-A), their composition into task-level strategies (Sec. IV-B), and strategy instantiation in novel scenes (Sec. IV-C). IV-A Primitives as Local Trajectory-Optimization Programs A primitive is a reusable behavior for realizing an object-level motion effect, such as pushing an object by a displacement or flipping an object by an angle. We represent a primitive as a local relational trajectory-optimization program Pi=(Gi,Ci,ĪØi,Ļi,Ji)ā.P_i=(G_i,C_i, _i, _i,J_i) . Here, GiG_i is a semantic, object-centric description of the intended effect; CiC_i is the relational interface; ĪØi _i is the verifiable success condition; Ļi _i is the geometry resolver; and JiJ_i is the optimization cost. Together, these components specify how the primitive is instantiated, optimized, and verified in a specific scene. A primitive is therefore neither a fixed trajectory nor a black-box policy. Table I illustrates each component for flipping an object against a fixture. TABLE I: Local trajectory-optimization program for the primitive flip. Component Content Semantic goal GiG_i Flip the target object against a fixture by a specific angle. Relational interface Ci=(i,Īi,i)C_i=(r_i, _i,q_i) Semantic roles ir_i: (target object, support plane, fixture, manipulator). Relations Īi _i: the object rests on the support plane, and the fixture exposes a vertical face oriented toward the target object and reachable by the manipulator. Effect parameter iq_i: the requested flipping angle. Success predicate ĪØi _i Check that the achieved rotation matches the flipping angle and that the target object remains supported, near the fixture, and settled. Geometry resolver Ļi _i Resolve the support plane, the vertical face of the fixture, the flipping axis, and the dimensions of the target object at each step. Final cost JifJ_i^f Penalize terminal rotation error relative to the flipping angle, the gap between the fixture and the target object, and terminal velocity. Progress cost JipJ_i^p Encourage rotation and manipulator proximity while discouraging backtracking and irregular motion. Relational Interface. We write the primitive interface as Ci=(i,Īi,i).C_i=(r_i, _i,q_i). Here, ir_i declares the semantic roles involved in the primitive; for example, for the primitive flip in Table I, these roles are (target object, support plane, fixture, manipulator). Īi _i specifies the relations among these roles; for flip, it encodes that the target object rests on the support plane and that the fixture exposes a vertical face oriented toward the target object and reachable by the manipulator. Finally, iq_i declares the effect parameter; for flip, it is the requested flipping angle. To apply a primitive in a scene, an invocation supplies runtime arguments (bi,gi)(b_i,g_i) satisfying biāBindEā”(i,Īi),giāValā”(i),b_i _E(r_i, _i), g_i (q_i), where BindEā”(i,Īi)Bind_E(r_i, _i) denotes the type-compatible assignments of the objects in the scene E to ir_i that satisfy Īi _i, and Valā”(i)Val(q_i) denotes the values matching the types declared by iq_i. Scene-dependent geometry remains internal to the primitive and is resolved by Ļi _i at execution time. Geometry Resolver. Let ā”(E)X(E) be the state space of environment E. For the state xtāā”(E)x_t (E) at time step t, the geometry resolver Ļi _i computes the geometric features Ļi,t _i,t: Ļi,t=Ļi(E,xt,bi),t=0,ā¦,T. _i,t= _i(E,x_t,b_i), t=0,ā¦,T. For example, for the primitive flip in Table I, the agent would write detectors for the support plane, the vertical face of the fixture, the flipping axis, etc. Success Predicate. For a terminal state xTāā”(E)x_T (E), the predicate ĪØiā(E,xT,gi,Ļi,T)ā0,1 _i(E,x_T;g_i, _i,T)ā\0,1\ provides an executable version of GiG_i. It checks the achieved object relations and requested effect, yielding an unambiguous signal for verification. Optimization Cost. For a simulated trajectory Ļ=(x0,ā¦,xT)Ļ=(x_0,ā¦,x_T) in environment E, where xtāā”(E)x_t (E), the primitive cost is Ji(E,Ļ;gi,Ļi,0:T)=Jif(E,xT;gi,Ļi,T)+Ī»iJip(E,Ļ;gi,Ļi,0:T). J_i(E,Ļ;g_i, _i,0:T)=J_i^f(E,x_T;g_i, _i,T)+ _iJ_i^p(E,Ļ;g_i, _i,0:T). Here, JifJ_i^f is the final cost, which measures how far xTx_T is from the effect encoded by GiG_i; JipJ_i^p is the progress cost, which may depend on the full trajectory and guides the optimizer through contact-rich interaction heuristics; and Ī»iā„0 _iā„ 0 is a constant. The coding agent proposes the heuristics by reasoning about the visual human demonstration. During primitive construction, the agent also proposes an initialization heuristic based on the resolved geometry and requested effect. The heuristic provides a plausible interaction trajectory from which to begin the search. Let z parameterize a candidate robot motion, let Ļā”(z)=RolloutEā”(x0,z)Ļ(z)=Rollout_E(x_0,z) denote its simulated state trajectory from x0x_0 in environment E, and let Ļi,0:T(z) _i,0:T(z) be the feature sequence resolved along that trajectory. The executable trajectory is obtained as ziā=argminzJi(E,Ļ(z);gi,Ļi,0:T(z)).z_i = _zJ_i\! (E,Ļ(z);g_i, _i,0:T(z) ). We solve this trajectory-optimization problem with the cross-entropy method (CEM) [56], a gradient-free sampling-based optimizer that allows us to optimize robot motions without differentiating through the contact-rich simulation. The primitive thereby adapts its motion to the object pose, size, geometry, physical properties, and requested effect. IV-B Strategies as Primitive Composition by Constraints Fig. 3: Composing primitives by relational constraints. After the primitives P are constructed, where each PiāP_i realizes an object-level motion effect, the next step is to compose them into a strategy Ī , a reusable task-level program that accomplishes an overall task. It specifies the sequence of primitives to invoke, how task roles map to each primitive, and how the requested effect of each invocation is determined: Ī =(GĪ ,CĪ ,ĪØĪ ,āØ(Pik,mk,ck)ā©k=1K), = (G_ ,C_ , _ , (P_i_k,m_k,c_k) _k=1^K ), where GĪ G_ describes the overall task goal and ĪØĪ _ evaluates its final outcome. Strategy Interface. A strategy exposes semantic roles and their required relations, CĪ =(Ī ,ĪĪ ).C_ =(r_ , _ ). Invoking the strategy requires a semantic role binding bāBindEā”(Ī ,ĪĪ )b _E(r_ , _ ) that assigns an entity to each strategy role while satisfying ĪĪ _ . Primitive effect parameters are not exposed through CĪ C_ ; instead, they are derived inside the strategy during execution. Primitive Composition by Constraints. A strategy consists of K phases, each represented by a tuple (Pik,mk,ck)(P_i_k,m_k,c_k), where PikāP_i_k is a primitive, mkm_k is a role map, and ckc_k is a connecting constraint. For the invoked primitive PikP_i_k, mkm_k produces a role binding bkb_k that fills the role declarations ikr_i_k, while ckc_k produces an effect argument gkg_k that fills the parameter declarations ikq_i_k. The connecting constraint declaratively specifies how phase k depends on the outcome of phase kā1k-1 and is evaluated just in time at the beginning of phase k: bk b_k =mkā(b)āBindEā”(ik,Īik), =m_k(b) _E(r_i_k, _i_k), gk g_k =ckā(E,skā1,b)āValā”(ik), =c_k(E,s_k-1,b) (q_i_k), where skā1āā”(E)s_k-1 (E) denotes the environment state after the execution of phase kā1k-1. The phase is then executed as sk=Execā”(Pik,E,skā1,gk,bk).s_k=Exec\! (P_i_k;E,s_k-1,g_k,b_k ). Here, ExecExec solves the local trajectory-optimization problem of PikP_i_k and executes the optimized trajectory. For example, Fig. 3 shows a press move following a flip. The role map mkm_k maps the strategyās target object, supporting surface, and tool roles to the corresponding roles of the press move primitive, producing bkb_k. The connecting constraint ckc_k determines the goal position from the relation between the brown obstacle and the white fixture. Using the target objectās current position in the post-flip state skā1s_k-1, it computes the displacement toward this goal along the fixture axis, producing the effect argument gkg_k for the press move primitive. IV-C Strategy Instantiation in Novel Scenes To deploy a strategy in a novel scene, we only need to provide a semantic binding. Given a novel scene EāE and instruction lāl , RAPID uses a coding agent to rapidly generate a semantic binding bāāBindEāā”(Ī ,ĪĪ )b _E (r_ , _ ) that assigns scene entities to the semantic roles exposed by the strategy interface. Since this step only associates the exposed semantic roles with scene entities, it is lightweight and incurs little deployment overhead (a few seconds). At phase k, the strategy derives the primitive binding and effect argument automatically. Throughout this process, the strategy program Ī and the referenced primitives remain frozen. Generalization therefore arises from the object-centric relational program representation, together with the adaptation of trajectory optimization to the geometry and physical properties of the new scene. V Agentic Programming from a Demonstration Algorithm 1 RAPID Input: Human demonstration D, instruction l Output: Verified primitives P and strategy Ī 1 (Di,li)i=1KāSegmentā(D,l)\(D_i,l_i)\_i=1^Kā Segment(D,l) 2 āā Pā 3 // Primitive construction ā³ Sec. V-B 4 for i=1,ā¦,Ki=1,ā¦,K do 5 E^iāReconstructā(Di) E_iā Reconstruct(D_i) ā³ Sec. V-A 6 (Gi,ĪØi)āInferObjectiveā(Di,li)(G_i, _i)ā InferObjective(D_i,l_i) 7 giāInferEffectā(Di,Gi)g_iā InferEffect(D_i,G_i) 8 PiāProposePrimitiveā(Di,Gi,ĪØi,E^i)P_iā ProposePrimitive(D_i,G_i, _i, E_i) 9 iā(gi,E^i)āŖVariantsā(gi,E^i,Pi)V_iā\(g_i, E_i)\āŖ Variants(g_i, E_i,P_i) 10 PiāVerifyReviseā(Pi,i,li)P_iā VerifyRevise(P_i,V_i,l_i) 11 āāŖFreezeā(Pi)P āŖ\ Freeze(P_i)\ 12 // Strategy composition ā³ Sec. V-C 13 E^0āReconstructā(D) E_0ā Reconstruct(D) 14 Ī āComposeStrategyā(D,l,,E^0) ā ComposeStrategy(D,l,P, E_0) 15 Ī āE^0āŖVariantsā(E^0,Ī )V_ ā\ E_0\āŖ Variants( E_0, ) 16 Ī āVerifyReviseā(Ī ,Ī ,l) ā VerifyRevise( ,V_ ,l) 17 return ,Freezeā(Ī )P, Freeze( ) 18 // Shared verification and revision 19 function VerifyRevise(Q,,a)(Q,V,a) 20 repeat 21 āāExecuteEvaluateā(Q,,a)Rā ExecuteEvaluate(Q,V,a) 22 if Ā¬AllSuccessfulā(ā) AllSuccessful(R) then 23 QāDiagnoseReviseā(Q,ā)Qā DiagnoseRevise(Q,R) 24 until AllSuccessfulā(ā) AllSuccessful(R) 25 return Q We now describe how RAPID constructs and verifies the programs defined in Sec. IV, covering environment reconstruction, primitive construction, and strategy composition. V-A Reconstructing Interactive Environments Interactive Environment. To verify a manipulation program, RAPID reconstructs an interactive simulation environment from the initial RGB-D frame of the demonstration. We first use a vision-language model (VLM) to identify the objects in the scene, assign semantic labels to them, and estimate their physical properties. Then, for each semantic name, we utilize SAM 3 [57] to get a segmentation mask. The RGB-D frame and the mask are then fed into SAM 3D [58] to reconstruct a complete mesh. Finally, we use FoundationPose [59] to register each mesh to the RGB-D frame and assemble the reconstructed objects and robot model in MuJoCo [60]. The resulting environment preserves the scene entities, geometry, and spatial relations needed to execute the candidate program and evaluate its success predicates. Feasible Scene Variants. To construct variants of the demonstrated scene, RAPID uses a task-agnostic variation sampler to randomize object appearances, sizes, poses, and physical parameters such as mass, friction, and elasticity. However, the randomly sampled scenes might not be feasible for a certain primitive or strategy. Therefore, we ask the coding agent to write a filter for each primitive or strategy that accepts only variants preserving the declared roles and relations, task meaning, and physical feasibility. V-B Primitive Construction Given the full demonstration D and its natural language description l, RAPID calls a subagent to partition D into an ordered sequence of interaction segments (D1,ā¦,DK)(D_1,ā¦,D_K) and assign a local description lil_i to each DiD_i. For each (Di,li)(D_i,l_i), if the coding agent recognizes that the behavior is covered by an existing primitive in the primitive library P, we can simply retrieve this primitive. Otherwise, the coding agent will infer a primitive specification consisting of a semantic goal GiG_i and a success predicate ĪØi _i from (Di,li)(D_i,l_i). It further reasons about the contact interactions that produce the demonstrated effect, the relational structure that should remain invariant across executions, and the quantities that might vary between invocations. For example, in Table I, the demonstrated rotation supplies the flipping-angle argument, while relations among the target object, support plane, fixture, and manipulator define the reusable interaction structure. Given this specification and interaction analysis, the coding agent generates the relational interface CiC_i, geometry resolver Ļi _i, and optimization cost JiJ_i introduced in Sec. IV-A. As shown in Algorithm 1, RAPID verifies and revises PiP_i across a set iV_i containing the demonstrated effectāenvironment pair (gi,E^i)(g_i, E_i) and feasible variants of both the environment and effect argument generated by a task-agnostic sampler. Verification continues until ĪØi _i is satisfied for every pair using its associated effect argument. The verified primitive is then added to the library P. V-C Strategy Composition After constructing the required primitives, RAPID composes them into a task-level strategy. As shown in Algorithm 1, given the full demonstration D and instruction l, the coding agent infers a strategy specification consisting of the overall task goal GĪ G_ and final success predicate ĪØĪ _ . The sequence of the interaction segments (D1,ā¦,DK)(D_1,ā¦,D_K) determines the ordering of the primitive composition. The agent then constructs the strategy interface CĪ C_ and, for each phase k, selects a verified primitive PikP_i_k, defines the role map mkm_k, and specifies the connecting constraint ckc_k. Together, these components form an initial version of the strategy Ī . RAPID verifies the strategy in the reconstructed environment E^0 E_0 and its feasible scene variants using the execution procedure in Sec. IV-B. Failures guide the coding agent to revise the composition while keeping the verified primitives fixed, until ĪØĪ _ is satisfied across this verification set Ī V_ . VI Experiments In this section, we conduct experiments showing that (1) RAPID successfully closes the agentic coding loop from a single visual human demonstration to construct reusable programs for nonprehensile manipulation (Table I); (2) our object-centric relational program representation improves the generalization of the constructed manipulation programs (Tables I and I); and (3) RAPID can be effectively deployed on a real-world robotic system (Fig. 5). TABLE I: Success rates on nonprehensile manipulation tasks in simulation. We report meanĀ± across three trials; the best results are bolded. Method Task 1 Task 2 Task 3 Task 4 Task 5 Task 6 Task 7 Task 8 Average CaP 0.053Ā±0.009Ā± 0.009 0.000Ā±0.000Ā± 0.000 0.000Ā±0.000Ā± 0.000 0.000Ā±0.000Ā± 0.000 0.000Ā±0.000Ā± 0.000 0.000Ā±0.000Ā± 0.000 0.013Ā±0.009Ā± 0.009 0.040Ā±0.016Ā± 0.016 0.013Ā±0.003Ā± 0.003 CaP + OReP 0.040Ā±0.000Ā± 0.000 0.000Ā±0.000Ā± 0.000 0.000Ā±0.000Ā± 0.000 0.000Ā±0.000Ā± 0.000 0.000Ā±0.000Ā± 0.000 0.000Ā±0.000Ā± 0.000 0.013Ā±0.009Ā± 0.009 0.033Ā±0.025Ā± 0.025 0.011Ā±0.002Ā± 0.002 CaP-Agent0 0.353Ā±0.115Ā± 0.115 0.040Ā±0.057Ā± 0.057 0.000Ā±0.000Ā± 0.000 0.013Ā±0.019Ā± 0.019 0.173Ā±0.176Ā± 0.176 0.053Ā±0.041Ā± 0.041 0.240Ā±0.128Ā± 0.128 0.293Ā±0.242Ā± 0.242 0.146Ā±0.025Ā± 0.025 CaP-Agent0 + SV 0.440Ā±0.141Ā± 0.141 0.053Ā±0.009Ā± 0.009 0.020Ā±0.028Ā± 0.028 0.047Ā±0.066Ā± 0.066 0.113Ā±0.034Ā± 0.034 0.080Ā±0.059Ā± 0.059 0.167Ā±0.066Ā± 0.066 0.087Ā±0.096Ā± 0.096 0.126Ā±0.043Ā± 0.043 RAPID w/o SV 0.933Ā±0.009Ā± 0.009 0.833Ā±0.125Ā± 0.125 0.400Ā±0.161Ā± 0.161 0.440Ā±0.315Ā± 0.315 0.447Ā±0.256Ā± 0.256 0.340Ā±0.236Ā± 0.236 0.567Ā±0.229Ā± 0.229 0.293Ā±0.415Ā± 0.415 0.532Ā±0.071Ā± 0.071 RAPID 0.933Ā±0.057Ā± 0.057 0.847Ā±0.025Ā± 0.025 0.700Ā±0.033Ā± 0.033 0.787Ā±0.082Ā± 0.082 0.793Ā±0.009Ā± 0.009 0.587Ā±0.050Ā± 0.050 0.807Ā±0.203Ā± 0.203 0.620Ā±0.028Ā± 0.028 0.759Ā±0.024Ā± 0.024 VI-A Experimental Setup for Nonprehensile Manipulation Tasks. In this section, we evaluate RAPID on eight nonprehensile manipulation tasks in simulation. For each task, we provide a single visual human demonstration together with a language description. In all tasks, the objects are initially not graspable by the parallel gripper, so the robot must execute a sequence of nonprehensile manipulation primitives, such as pushing, flipping, pivoting, and toppling, to reach the target configuration. Importantly, the primitives are not provided to the coding agent a priori. We provide the descriptions of the 8 tasks below, and Fig. 4 illustrates them in the real world: Fig. 4: Real-world execution of the solution trajectories found by RAPID. T1. Push the object toward the corridor, switch the contact face, and push it again to store it inside the corridor. T2. Push the object toward the fixture, flip it to expose a graspable edge, and grasp it. T3. Move the object away from the obstacles, push it toward the fixture, flip it to expose a graspable edge, and grasp it. T4. Push the object toward the fixture, flip it, reposition it to expose a graspable edge, and grasp it. T5. Push the object toward the fixture, flip it to align with the opening, and move it into the opening for storage. T6. Pivot the box to align it with the corridor, push it toward the corridor, and then move it to store it inside the corridor. T7. Topple the object to expose an edge and grasp it. T8. Move the object to the side of the platform to expose a graspable edge, and grasp it. Benchmark. We construct a simulation benchmark in MuJoCo [60] using a Franka Research 3 robot arm equipped with a compliant operational-space controller [61]. Our primary evaluation metric is generalization. For each of the 8 tasks, we construct 50 novel test scenes that differ from the demonstration scene in object pose, appearance, size, shape, and material. We also introduce irrelevant objects to make the test scenes visually distracting. We evaluate each method by measuring its success rate on each task using a private success criterion hidden from RAPID. We run three trials of each experiment and report the mean and standard deviation. Baselines. We compare six approaches along three principal axes: the closed agentic loop (Agent), the object-centric relational program representation (OReP), and synthesized scene variants (SV). 1) Open-loop + Python (CaP): For each novel test scene, we use Code as Policies (CaP) [9] to generate a Python program conditioned on the demonstration, language description, and observation, without simulation feedback during program generation. 2) Open-loop + OReP (CaP + OReP): We augment CaP with our OReP and cross-entropy method (CEM) optimizer, enabling motions to be expressed relative to scene entities. 3) Agent + Python (CaP-Agent0): We use CaP-Agent0 from CaP-X [6] to assess the benefit of our program representation. We give CaP-Agent0 a more favorable agentic-programming setting than that of RAPID: during program construction, it receives rollout feedback consisting of ground-truth success signals, environment states, and visual observations; at evaluation time, it may additionally execute rollouts in each test scene and revise its program. 4) Agent + Python + SV (CaP-Agent0 + SV): We further augment CaP-Agent0 with our scene-variant generation module, allowing it to refine its program on synthesized scene variants. 5) Agent + OReP (RAPID w/o SV): We remove the scene-variant generation module from RAPID. 6) Agent + OReP + SV (RAPID): Our full method. Comparisons of 4) vs. 3) and 6) vs. 5) isolate the benefit of variant-based verification. We do not compare with ASPIRE [5] on nonprehensile tasks because it requires motion primitives to be provided. We use Codex [1] powered by GPT-5.6 Sol with high reasoning effort as the coding agent and provide the same robot control APIs for all methods. VI-B Simulation Experiments for Nonprehensile Manipulation RAPID successfully constructs reusable programs by closing the agentic loop from a single demonstration. As shown in Table I, vanilla CaP performs poorly despite being allowed to generate a program tailored to each test scene. Without an interactive environment for verifying its programs, the agent cannot check whether the proposed motions produce the intended object interactions. Augmenting CaP with our object-centric relational program representation does not improve its performance either, suggesting that the representation alone is insufficient: without closed-loop verification, the coding agent struggles to reason about object relations and therefore fails to construct meaningful relational programs. In contrast, RAPID reconstructs a simulation environment from the demonstration, infers a success predicate from the demonstration and language instruction, and uses this predicate to verify and iteratively refine candidate programs in simulation. In this way, RAPID constructs a reusable program for each task that generalizes across diverse test scenes. In our experiments, this construction takes on the order of tens of minutes per task. This cost is incurred once per task: the resulting strategy and primitives remain fixed across the 50 novel scenes, with semantic role binding taking only a few seconds per scene. The object-centric relational program representation enhances the generalization of the strategy program. The comparison with the two CaP-Agent0 baselines in Table I supports the benefit of our program representation: even with stronger verification access, including ground-truth feedback and evaluation-time refinement, CaP-Agent0 substantially underperforms RAPID. Although CaP-Agent0 incorporates state feedback, its generated policies often use fixed world-frame offsets and predetermined timing, limiting spatial generalization and robustness to state deviations during execution. In contrast, RAPID computes geometry-dependent subgoals from the state reached after each preceding phase. The gap is even larger for contact-rich, forceful behaviors such as flipping and toppling, which are naturally expressed by our relational, optimization-based motion primitives. Ablation study of scene variant generation. To evaluate the effectiveness of scene variant generation, we remove it from RAPID and report the results in Table I. Except for Tasks 1 and 2, the performance of RAPID drops substantially, while the standard deviation of the success rate increases, indicating reduced stability and robustness. Interestingly, augmenting CaP-Agent0 with scene variant generation does not improve its performance. This suggests that, without our object-centric relational program representation, CaP-Agent0 struggles to construct a generalizable program that can solve diverse generated variants without overfitting to individual scenes. TABLE I: Experimental Results on LIBERO-Pro. Method libero-object libero-goal libero-spatial Pos. (Avg.) Task. (Avg.) Pos. (Avg.) Task. (Avg.) Pos. (Avg.) Task. (Avg.) OpenVLA 0 0 0 0 0 0 Ļ0 _0 0 0 0 0 0 0 Ļ0.5 _0.5 0.17 0.01 0.38 0 0.20 0.01 CaP-Agent0 0.22 0.18 0.26 0.17 0.12 0.14 ASPIRE 0.98 0.95 0.81 0.45 0.51 0.60 RAPID 1 0.95 0.67 0.53 0.83 0.81 VI-C Simulation Experiments on LIBERO-Pro Experimental Setup. To demonstrate the general applicability of RAPID beyond nonprehensile manipulation, we evaluate it on LIBERO-Pro [18], an extension of LIBERO [62] with stronger position (Pos.) and instruction (Task.) perturbations. LIBERO-Pro predominantly features prehensile tasks, including pick-and-place and grasp-based articulated-object manipulation. We compare against two groups of baselines: (1) vision-language-action (VLA) models, including OpenVLA [63], Ļ0 _0 [64], and Ļ0.5 _0.5 [65], whose results are directly taken from the LIBERO-Pro paper; and (2) code-as-policy (CaP) methods, including CaP-Agent0 [6] and ASPIRE [5]. For a fair comparison with CaP methods, which do not use demonstrations, RAPID also receives no demonstrations and is given only the task description and a single debug scene. This setting is more restrictive than the setting used by ASPIRE, which uses 15 debug scenes per task. Moreover, while CaP baselines are provided with predefined motion primitives such as grasping, placing, and transporting, RAPID must discover and implement these primitives solely through exploration and iterative refinement in the debug scene. Results. Table I reports the results on LIBERO-Pro. RAPID achieves the highest success rate on five of the six task settings, demonstrating the effectiveness of our object-centric relational program representation. On LIBERO-Goal (Pos.), RAPID ranks second behind ASPIRE. This gap stems from articulated-object manipulation: primitives discovered by RAPID from only a single debug scene, such as drawer opening, are less robust to spatial variation, whereas ASPIRE is provided with an interpolate segment motion primitive that simplifies such interactions. Overall, these results demonstrate the generality of our framework beyond nonprehensile manipulation. VI-D Real-World Experiments for Nonprehensile Manipulation To demonstrate that RAPID can operate effectively on a real robotic system, we conduct real-world experiments on the eight nonprehensile manipulation tasks shown in Fig. 4. As illustrated in Fig. 5, our setup consists of a Franka Research 3 robot arm equipped with a parallel gripper and an Intel RealSense L515 camera for RGB-D observations. We compare RAPID against CaP-Agent0 [6]. For each task, we evaluate 10 test scenes and report the success rate. Results. As shown in Fig. 5, with our object-centric relational program representation and compliant operational-space controller, contact-rich nonprehensile behaviors optimized in simulation can be transferred directly to the real robot and executed effectively. In contrast, CaP-Agent0 struggles to generate reliable trajectories using procedural programs. Moreover, leveraging the VLMās prior knowledge to estimate physical parameters such as mass, friction, and restitution, together with SAM 3Dās strong 3D reconstruction capability, helps keep the real-to-sim and sim-to-real gaps manageable. Fig. 5: Real-world experimental setup (left) and results (right). VII Conclusion We presented RAPID, a framework that closes the agentic coding loop for robotic manipulation from a single visual human demonstration and a language description. By automatically deriving task specifications, manipulation primitives, and interactive verification environments, RAPID enables a coding agent to construct, execute, and iteratively refine reusable manipulation strategies. Our object-centric relational program representation represents manipulation primitives as trajectory-optimization programs and composes them with relational constraints, enabling generalization to novel scenes. Experiments on eight contact-rich nonprehensile manipulation tasks demonstrate strong generalization across object and scene variations, while evaluations on LIBERO-Pro and a real robotic system support RAPIDās broader applicability and practical deployment. In future work, RAPID will incorporate other tools for learning primitives, such as reinforcement learning for manipulation with dexterous hands. Further, RAPID will automatically benefit from the relentless progress in coding agents, robot foundation models, and simulation engines (e.g., for deformable objects [66]), and scale up. References [1] OpenAI, āCodex,ā 2025, website: https://openai.com/codex/. [2] Anthropic, āClaude Code,ā 2025, website: https://claude.com/product/claude-code. [3] K. Chen et al., āGaP: A Graph-as-Policy Multi-Agent Self-Learning Harness For Variational Automation Tasks,ā arXiv:2607.05369, 2026. [4] W. Xiao et al., āENPIRE: Agentic Robot Policy Self-Improvement in the Real World,ā arXiv:2606.19980, 2026. [5] R. Lu et al., āASPIRE: Agentic /Skills Discovery for Robotics,ā arXiv:2607.00272, 2026. [6] L. Fu et al., āCaP-X: A framework for benchmarking and improving coding agents for robot manipulation,ā in ICML, 2026. [7] S. Berman, M. Ilie, J. Deng, and D. Freeman, āClaude Plays Robotics,ā 2026, research Blog: https://w.anthropic.com/research/claude-plays-robotics. [8] Noematrix Team, āRoboRSI: Stable, Efficient, and Reusable Robot Self-Evolution in Complex Real-World Environments,ā 2026, research Blog: https://lab.noematrix.ai/blog/2-roborsi/. [9] J. Liang et al., āCode as Policies: Language Model Programs for Embodied Control,ā in ICRA, 2023. [10] A. Ng, āThree Key Loops for Building Great Software,ā 2026, research Blog: https://w.deeplearning.ai/the-batch/three-key-loops-for-building-great-software. [11] D. Huang et al., āAgentCoder: Multi-Agent-Based Code Generation with Iterative Testing and Optimisation,ā arXiv:2312.13010, 2023. [12] C. R. Garrett et al., āIntegrated Task and Motion Planning,ā Annual Review of Control, Robotics, and Autonomous Systems, vol. 4, p. 265ā293, 2021. [13] M. T. Mason, Mechanics of Robotic Manipulation. MIT Press, 2001. [14] A. Holladay, R. Paolini, and M. T. Mason, āA General Framework for Open-Loop Pivoting,ā in ICRA, 2015. [15] X. Cheng, E. Huang, Y. Hou, and M. T. Mason, āContact Mode Guided Sampling-Based Planning for Quasistatic Dexterous Manipulation in 2D,ā in ICRA, 2021. [16] L. U. Odhner, R. R. Ma, and A. M. Dollar, āOpen-Loop Precision Grasping with Underactuated Hands Inspired by a Human Manipulation Strategy,ā T-ASE, vol. 10, no. 3, p. 625ā633, 2013. [17] X. Cheng, E. Huang, Y. Hou, and M. T. Mason, āContact Mode Guided Motion Planning for Quasidynamic Dexterous Manipulation in 3D,ā in ICRA, 2022. [18] X. Zhou et al., āLIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models beyond Memorization,ā arXiv:2510.03827, 2025. [19] I. Singh et al., āProgPrompt: Generating Situated Robot Task Plans using Large Language Models,ā in ICRA, 2023. [20] C. Wang et al., āChain-of-Modality: Learning Manipulation Programs from Multimodal Human Videos with Vision-Language-Models,ā in ICRA, 2025. [21] W. Huang et al., āVoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models,ā in CoRL, 2023. [22] W. Huang, C. Wang, Y. Li, R. Zhang, and F.-F. Li, āReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation,ā in CoRL, 2024. [23] Y. J. Ma et al., āEureka: Human-Level Reward Design via Coding Large Language Models,ā in ICLR, 2024. [24] T. Xie et al., āText2Reward: Reward Shaping with Language Models for Reinforcement Learning,ā in ICLR, 2024. [25] Y. Wang et al., āRoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulation,ā in ICML, 2024. [26] R. Li et al., āRoboClaw: An Agentic Framework for Scalable Long-Horizon Robotic Tasks,ā arXiv:2603.11558, 2026. [27] G. Wang et al., āVoyager: An Open-Ended Embodied Agent with Large Language Models,ā TMLR, 2024. [28] J. Zhang et al., āPlayful Agentic Robot Learning,ā arXiv:2606.19419, 2026. [29] E. Johns, āCoarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration,ā in ICRA, 2021. [30] B. Wen, W. Lian, K. Bekris, and S. Schaal, āYou Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstration,ā in RSS, 2022. [31] A. Simeonov et al., āNeural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation,ā in ICRA, 2022. [32] W. Shen et al., āDistilled Feature Fields Enable Few-Shot Language-Guided Manipulation,ā in CoRL, 2023. [33] J. Zhu et al., āDensematcher: Learning 3d semantic correspondence for category-level manipulation from a single demo,ā in ICLR, 2025. [34] O. Biza et al., āOne-shot Imitation Learning via Interaction Warping,ā in CoRL, 2023. [35] Y. Liu, J. Mao, J. B. Tenenbaum, T. Lozano-PĆ©rez, and L. P. Kaelbling, āOne-Shot Manipulation Strategy Learning by Making Contact Analogies,ā in ICRA, 2025. [36] M. Sieb, Z. Xian, A. Huang, O. Kroemer, and K. Fragkiadaki, āGraph-Structured Visual Imitation,ā in CoRL, 2019. [37] Y. Zhu, A. Lim, P. Stone, and Y. Zhu, āVision-Based Manipulation from Single Human Video with Open-World Object Graphs,ā Autonomous Robots, vol. 50, no. 2, p. 27, 2026. [38] Z. Zhang et al., āUniversal Visual Decomposer: Long-Horizon Manipulation Made Easy,ā in ICRA, 2024. [39] T. Gao, S. Nasiriany, H. Liu, Q. Yang, and Y. Zhu, āPRIME: Scaffolding Manipulation Tasks with Behavior Primitives for Data-Efficient Imitation Learning,ā RA-L, vol. 9, no. 10, p. 8322ā8329, 2024. [40] A. Wu, R. Wang, S. Chen, C. Eppner, and C. K. Liu, āOne-shot transfer of long-horizon extrinsic manipulation through contact retargeting,ā in IROS, 2024. [41] W. Liu, N. Nie, R. Zhang, J. Mao, and J. Wu, āLearning Compositional Behaviors from Demonstration and Language,ā in CoRL, 2024. [42] S. Cheng, C. R. Garrett, A. Mandlekar, and D. Xu, āNOD-TAMP: Generalizable Long-Horizon Planning with Neural Object Descriptors,ā in CoRL, 2024. [43] N. Nie et al., āLearning Composable Skills by Discovering Spatial and Temporal Structure with Foundation Models,ā in ICRA, 2026. [44] B. Zandonati, T. Lozano-PĆ©rez, and L. P. Kaelbling, āRational Inverse Reasoning: Few-Shot Imitation by Inferring Intent through Planning,ā arXiv:2508.08983, 2025. [45] N. Chavan-Dafle et al., āExtrinsic Dexterity: In-Hand Manipulation with External Forces,ā in ICRA, 2014. [46] K. M. Lynch and M. T. Mason, āStable Pushing: Mechanics, Controllability, and Planning,ā IJRR, vol. 15, no. 6, p. 533ā556, 1996. [47] G. Lee, T. Lozano-PĆ©rez, and L. P. Kaelbling, āHierarchical Planning for Multi-Contact Non-Prehensile Manipulation,ā in IROS, 2015. [48] J.-P. Sleiman, J. Carius, R. Grandia, M. Wermelinger, and M. Hutter, āContact-Implicit Trajectory Optimization for Dynamic Object Manipulation,ā in IROS, 2019. [49] B. Aceituno-Cabezas and A. Rodriguez, āA Global Quasi-Dynamic Model for Contact-Trajectory Optimization in Manipulation,ā in RSS, 2020. [50] T. Pang, H. J. T. Suh, L. Yang, and R. Tedrake, āGlobal Planning for Contact-Rich Manipulation via Local Smoothing of Quasi-Dynamic Contact Models,ā T-RO, 2023. [51] E. Huang, X. Cheng, Y. Mao, A. Gupta, and M. T. Mason, āAutogenerated Manipulation Primitives,ā IJRR, 2023. [52] W. Zhou and D. Held, āLearning to Grasp the Ungraspable with Emergent Extrinsic Dexterity,ā in CoRL, 2022. [53] W. Zhou, B. Jiang, F. Yang, C. Paxton, and D. Held, āHACMan: Learning Hybrid Actor-Critic Maps for 6D Non-Prehensile Manipulation,ā in CoRL, 2023, p. 241ā265. [54] Y. Cho, J. Han, Y. Cho, and B. Kim, āCORN: Contact-based Object Representation for Nonprehensile Manipulation of General Unseen Objects,ā in ICLR, 2024. [55] S. Nasiriany, H. Liu, and Y. Zhu, āAugmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Tasks,ā in ICRA, 2022. [56] P.-T. de Boer, D. P. Kroese, S. Mannor, and R. Y. Rubinstein, āA Tutorial on the Cross-Entropy Method,ā Annals of Operations Research, vol. 134, no. 1, p. 19ā67, 2005. [57] N. Carion et al., āSAM 3: Segment Anything with Concepts,ā in ICLR, 2026. [58] X. Chen et al., āSAM 3D: 3Dfy Anything in Images,ā in CVPR, 2026. [59] B. Wen, W. Yang, J. Kautz, and S. Birchfield, āFoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects,ā in CVPR, 2024. [60] E. Todorov, T. Erez, and Y. Tassa, āMuJoCo: A Physics Engine for Model-Based Control,ā in IROS, 2012. [61] O. Khatib, āA Unified Approach for Motion and Force Control of Robot Manipulators: The Operational Space Formulation,ā IEEE Journal on Robotics and Automation, vol. 3, no. 1, p. 43ā53, 1987. [62] B. Liu et al., āLIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning,ā in NeurIPS, 2023. [63] M. J. Kim et al., āOpenVLA: An Open-Source Vision-Language-Action Model,ā in CoRL, 2024. [64] K. Black et al., āĻ0 _0: A Vision-Language-Action Flow Model for General Robot Control,ā in RSS, 2025. [65] Physical Intelligence et al., āĻ0.5 _0.5: A Vision-Language-Action Model with Open-World Generalization,ā arXiv:2504.16054, 2025. [66] NVIDIA, āNewton Physics Engine,ā 2025, website: https://developer.nvidia.com/newton-physics. IEEEcontrol