Paper deep dive
SUN: Persistent Programs For Language-Grounded Control-to-Learning-to-Real Policies
Weiqi Wang, Zhi Li, Yudong Lei, David Martinez, Xiaofeng Gao, Yuxin Jiang, Chenfanfu Jiang, Yingnian Wu, Demetri Terzopoulos, Ran Gong
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Bridging model-based control and learned policies in long-horizon manipulation has harbored a silent disagreement: control executes specified objectives, learning amortizes that behavior into a reactive policy, yet existing protocols discard task semantics, leaving rewards hand-crafted and behavior drifting from what control this http URL introduce Semantically UNified (SUN) Programs, typed executables where geometric and contact relations are defined once and compiled into aligned Model Predictive Control (MPC) costs, satisfaction predicates, RL rewards, transition guards, and diagnostics. Our system, Kuafu, driven by large vision language systems, automatically synthesizes SUN Programs from language and scene semantics, screens feasibility via MPC, and retains semantics while training stage-conditioned policies. Across nine tasks, Kuafu achieves 82.03% macro-success, outperforming sparse-reward (35.67%) and Stage-BC (24.75%) baselines. At 8192-way scale, it generates 10.57x the successful trajectory time per hour of human teleoperation. With 500 trajectories per task, Kuafu data trains DP3 policies to 46.0% simulation success (vs. 22.4% for alternatives) and 34.7% on physical Franka and Kinova robots. These results establish that simulation-screened task semantics can effectively amortize control into robust policies, without demonstrations or manual dense rewards, unifying symbolic planning and data-driven execution.
Tags
Links
- Source: https://arxiv.org/abs/2608.31167v1
- Canonical: https://arxiv.org/abs/2608.31167v1
Trouble viewing inline? Open PDF directly ā
Full Text
55,971 characters extracted from source content.
Expand or collapse full text
SUN: Persistent Programs For Language-Grounded Control-to-Learning-to-Real Policies Thanks: *Equal contribution.Thanks: ā Corresponding author.Thanks: ā”This work does not relate to the authorās position at Amazon. Weiqi Wang Affiliation: University of California, Los Angeles Zhi Li Affiliation: University of California, Los Angeles Yudong Lei Affiliation: University of California, Santa Barbara David Martinez Affiliation: University of California, Los Angeles Xiaofeng Gao Affiliation: Amazon Yuxin Jiang Affiliation: University of California, Los Angeles Chenfanfu Jiang Affiliation: University of California, Los Angeles Yingnian Wu Affiliation: University of California, Los Angeles Demetri Terzopoulos Affiliation: University of California, Los Angeles Ran Gong Affiliation: Robotics and AI Institute Abstract Bridging model-based control and learned policies in long-horizon manipulation has harbored a silent disagreement: control executes specified objectives, learning amortizes that behavior into a reactive policy, yet existing protocols discard task semantics, leaving rewards hand-crafted and behavior drifting from what control verified. We introduce Semantically UNified (SUN) Programs, typed executables where geometric and contact relations are defined once and compiled into aligned Model Predictive Control (MPC) costs, satisfaction predicates, RL rewards, transition guards, and diagnostics. Our system, Kuafu, driven by large vision language systems, automatically synthesizes SUN Programs from language and scene semantics, screens feasibility via MPC, and retains semantics while training stage-conditioned policies. Across nine tasks, Kuafu achieves 82.03% macro-success, outperforming sparse-reward (35.67%) and Stage-BC (24.75%) baselines. At 8192-way scale, it generates 10.57Ć the successful trajectory time per hour of human teleoperation. With 500 trajectories per task, Kuafu data trains DP3 policies to 46.0% simulation success (vs. 22.4% for alternatives) and 34.7% on physical Franka and Kinova robots. These results establish that simulation-screened task semantics can effectively amortize control into robust policies, without demonstrations or manual dense rewards, unifying symbolic planning and data-driven execution. Index TermsāIntegrated Planning and Learning, Manipulation Planning, Optimization and Optimal Control, Imitation Learning, Datasets for Robot Learning I Introduction Reinforcement learning (RL) is an attractive method for multi-stage manipulation because it can learn high-dimensional, contact-rich behaviors directly from interaction without requiring an accurate model of the underlying dynamics [32]. However, RL requires dense, structured rewards to expose intermediate progress, yet manually crafting these signals is laborious and brittle. Minor adjustments to reward terms or transition thresholds often yield drastically different behaviors, demanding extensive trial-and-error tuning. While recent methods automate reward generation [13], they suffer from a critical latency: the impact of a candidate reward can only be evaluated after full policy training and rollout. Consequently, each iterative revision incurs the full cost of a training cycle, rendering the search over reward structures inefficient. Model-based planning and control explicitly encode task structures, defining geometric objectives and transition logic to enable testing and revision under closed-loop dynamics [24, 16]. However, this approach is highly sensitive to model fidelity; successful execution relies on accurate state reconstruction, precise interaction models, and strict adherence to task definitions. In practice, even minor discrepancies, such as model mismatch, observation noise, or specification errors, can cause controller failure. Furthermore, constructing these robust formulations demands substantial manual engineering for every new task [4]. Fig. 1: Kuafu preserves the SUN Program across language grounding, MPC feasibility screening, policy learning, and monitored data generation. Downstream visual-policy training and zero-shot deployment rely exclusively on the generated observations and actions, unaware of the original program. These complementary properties have motivated efforts to bridge model-based control and learning. Most prominently, model-based controllers are used to generate successful trajectories for policy training [11, 4]. While this transfers physically grounded behavior, it discards the objectives, constraints, stage transitions, or satisfaction boundaries that defined why those actions were correct. Although recent methods expose richer task structure to control and learning [18, 8], the representations are often recast as distinct learning signals, forcing stage or completion logic to be re-implemented. As a result, the rewards guiding the policy may drift from the specification validated by control, allowing inconsistencies to surface only after deployment. Fig. 2: Kuafu grounds language instruction into a typed SUN Program over scene frames, screens feasibility with simulated physics via MPC, and retains it across policy learning and data production. Validated MPC rollouts and program annotations supervise behavioral cloning, while the retained program provides transitions, residual rewards, calibration, and diagnostics to the controller, which then generates sensor-action data for visual policies. We argue that a persistent task program that survives closed-loop screening in simulated physics can significantly enhance multi-stage robot learning. To this end, we introduce the Semantically UNified (SUN) Program, a language-grounded, typed executable artifact. Given a language instruction, an LLM agent constructs each SUN Program from system-provided scene frames and typed operators, binding their arguments and organizing the geometric relations into ordered stages. Each declared relation is coupled with a satisfaction boundary, and compiled into transition guards, diagnostic view, and aligned objectives for both Model Predictive Control (MPC) and Residual RL. Its end-to-end implementation, Kuafu 11 1 Kuafu is a figure in Chinese mythology who chased the sun; our system likewise pursues a persistent SUN Program across control, learning, and deployment. (Figure 1), retains the accepted artifact alongside a stage-conditioned whole-task policy. Optimization thus serves not merely as a trajectory generator, but as an efficient screener of task knowledge, rejecting faulty objectives and transitions before they incur repeated policy-training cycles. Kuafu then uses the screened SUN supervision to learn contact-rich execution through Residual RL, retaining controlās ability to validate task structure while replacing model-dependent online optimization with a deployable policy. Across nine manipulation tasks, we demonstrate that the value of the persistent program scales with task horizon and cannot be reduced to mere behavior transfer. Compared with reward-learning and online-planning baselines, Kuafu achieves higher success rates with fewer language-model interactions and at a fraction of the computational cost. Furthermore, the persistent program guides the efficient collection of simulation data for sim-to-real learning, yielding significantly higher downstream policy success than comparable data-generation methods. Finally, we validate these results through sim-to-real transfer on physical Franka and Kinova robots, confirming the robustness of our approach in real-world settings. Our contributions are threefold: ā¢ We introduce Semantically UNified (SUN) Programs, a typed executable artifact that preserves task semantics across control, learning, and deployment, eliminating the semantic drift common in current handoffs. ā¢ We present Kuafu, a system that synthesizes SUN Programs from natural language, screens them via MPC, and trains stage-conditioned policies without manual reward engineering or seed demonstrations. ā¢ We demonstrate that Kuafu achieves higher success rates with fewer language-model interactions and at a fraction of the computational cost of competing methods, while enabling robust zero-shot sim-to-real transfer on physical robots. I Related Work Executable Task Representations. Long-horizon manipulation requires representations connecting discrete task progression with continuous motion. Symbolic planning and program generation couple action sequences to geometric feasibility or scene grounding [24, 22]. Language-grounded optimization converts task descriptions into composable 3D value maps or reward parameters evaluated by MPC, exposing spatial objectives to closed-loop control [9, 28]. Formal approaches encode temporal progress as finite-state rewards or signal-temporal-logic constraints for learning and predictive control [23, 17]. While these representations constrain search and operationalize task progress, they generally remain confined to a single planning, control, or learning stage, breaking semantic continuity across the pipeline. Consequently, a state accepted by one component may still produce infeasible refinements, premature handoffs, or misleading learning signals that compound over long-horizon execution. Control-to-Policy Transfer. Task structure enters policy learning through generated objectives or controller-derived supervision. Language models reduce manual design by generating executable rewards and, for sim-to-real transfer, domain-randomization configurations [26, 13, 14]. Since iterated objectives can overfit observed behavior or encode invalid task specifications, a high return therefor may not translate into strong performance on the true task metric [2]. Complementarily, optimized trajectories provide supervised policy targets, while residual learning preserves nominal control and learns corrective actions [12, 10]. Iterative rewards generation are flexible, but each revision costs a full reinforcement-learning trial and another round of language model query. We instead restrict language models to executable geometric relations that control method can screen under close loop physics, rejecting invalid specifications at a fraction of the interaction cost. Autonomous Robot Data Generation. Data generators scale robot experience by retargeting object-relative demonstration segments across scene configurations and decomposing behavior into reusable skills connected by planned motions [15, 5]. These convert sparse demonstrations into diverse imitation data, yet task semantics remain bound to the originals, lacking an independent, grounded monitor to validate learned-policy rollouts. The closest precedents span three complementary interfaces. ReKep [8] expresses multi-stage tasks as sequences of grounded keypoint cost functions optimized in closed loop, while OmniManip [19] maps object-centric interaction primitives to 3D spatial constraints with high-level VLM checking and low-level pose tracking. LATO [11] adapts an MPC teacher to the state distribution induced by the learner, whereas MPC-Net [4] trains a policy with a Hamiltonian loss encoding the original control objective and constraints. RoboGen [25] and GenSim2 [7] automate simulation task and scene construction together with skill or demonstration generation through foundation models, optimization, planning, and RL. Kuafu unifies these interfaces by enforcing a persistent, MPC-screened representation where each relationās identity, grounding, and satisfaction boundary remain fixed from controller validation through policy acquisition, learned execution, and monitored data production. I Methods We consider a long-horizon manipulation task described by a user-provided natural-language instruction āI. Kuafu uses an LLM to construct a SUN Program Ī from registered scene frames ā±F and the typed operator library āL. After closed-loop screening in simulated physics, the accepted program provides compiled supervision for learning a stage-conditioned whole-task controller Ļ. At time t, Ļ maps the current observation oto_t and active stage ztz_t to a robot action utu_t, such that ut=Ļā”(ot,zt)u_t=Ļ(o_t,z_t). Closed-loop execution of Ļ completes the task stages and generates sensorāaction trajectories for downstream visual-policy learning. We assume access to registered scene semantics ā±F, containing the identities and spatial and geometric information of the objects, and a typed operator library āL that defines reusable relations over this information. Kuafu obtains Ļ by constructing a SUN Program Ī from (ā,ā±,ā)(I,F,L), screening its execution through MPC, and using the successful rollouts to train the Stage-BC policy ĻĪø _Īø. The retained program then supplies stage context and program-derived learning signals to train the residual policy ĻĻ _Ļ; the bounded composition of ĻĪø _Īø and ĻĻ _Ļ defines the whole-task policy Ļ. We define a lineage as the coherent chain linking an accepted SUN Program, its MPC-generated trajectories, the Stage-BC prior, the bounded residual controller, and the final evaluated policy. Figure 2 outlines the life cycle of a SUN Program within Kuafu. I-A Semantically UNified (SUN) Programs Central to Kuafu is the SUN Program Ī , a typed, executable representation of the task specified by āI. Kuafu uses an LLM agent to construct Ī by selecting operators from the system-provided library āL, binding them to registered frames in ā±F, and organizing the grounded relations into stages according to āI. The LLM therefore composes existing typed components rather than inventing new frames, relations, or implementations. Each relation is defined once and compiled into aligned interfaces for MPC, stage monitoring, and reinforcement learning. The resulting program is retained across screening, policy learning, and data generation. Consider the instruction āopen the drawer.ā The scene registry ā±F provides the location of the drawerās base, handle and individual drawer, as well as the orientation and operation direction in terms of axis (handle bar along x-axis, +y for pushing in). āL provides typed relations such as āalign_axisā, ādirection_moveā, āfix_orientationsā. We use the drawer-opening portion of this task as a running example below. Semantic formation via LLM. For long-horizon tasks, we define the augmented state as x~t=(xt,zt,Ī¾zt) x_t=(x_t,z_t, _z_t), where ztz_t is the active stage and Ī¾k _k records physical snapshots (e.g., frame poses, gripper state) captured upon entry to stage k. The robot state, observation, and action are denoted by xtx_t, oto_t, and utu_t, respectively. Each operator schema in the library āL declares typed arguments, a canonical residual function, units, stage-entry snapshot behavior, a satisfaction tolerance, and a default reward weight. Kuafu forms the SUN Program by selecting schemas from āL and binding their arguments to registered frames in ā±F and task-specific scalar parameters: Ī=Īkk=1K,Īk=(ek,j(x,Ī¾k),Ļµk,j,wk,j)j=1mk. gathered =\ _k\_k=1^K, _k= \ (e_k,j(x, _k), _k,j,w_k,j ) \_j=1^m_k. gathered (1) Here, K is the number of stages and mkm_k is the number of grounded relations in stage k. Each term contains a grounded physical relation represented by ek,je_k,j, its satisfaction tolerance Ļµk,j _k,j, and its relative learning weight wk,jw_k,j. For each stage, the compiler maps every grounded relation to three executable interfaces: Compileā”(Īk)=(Ļk,jmpc,Pk,j,ck,jrl)j=1mk.Compile( _k)= \ (Ļ^mpc_k,j,P_k,j,c^rl_k,j ) \_j=1^m_k. (2) The MPC term Ļk,jmpcĻ^mpc_k,j contributes to the stage objective, the predicate Pk,jP_k,j is evaluated by the stage monitor, and the learning term ck,jrlc^rl_k,j contributes to the reinforcement-learning signal. Their exact numerical forms are operator-specific, but all three retain the same relation identity, grounding, and stage membership. For example, to pull the drawer open, the relation ādirectional_moveā can measure the drawer displacement relative to the pose stored in Ī¾k _k when the stage begins. Its compiled MPC term penalizes the remaining displacement, Pk,jP_k,j tests the same residual against Ļµk,j _k,j, and ck,jrlc^rl_k,j supplies the corresponding learning cost. An additional āfix_orientationā will measure the toolās angular displacement relative to the snapshot, keeping the gripās direction steady. This typing reduces synthesis complexity to verified operator selection and typed argument binding. We refer to this correspondence across control, monitoring, and learning as Semantic Unification. Stage progression. At runtime, Ī also defines the stage monitor. Each Īk _k contains the relations evaluated during stage k, and all of their predicates participate in stage completion. After applying utu_t, the monitor evaluates the post-action augmented state x^t+1=(xt+1,zt,Ī¾zt) x_t+1=(x_t+1,z_t, _z_t) before updating the active stage. We use DkD_k to denote that every predicate Pk,jP_k,j in stage k is satisfied. The program advances from ztz_t to zt+1z_t+1 when DztD_z_t is true; otherwise the current stage remains active. When a new stage begins, its entry-scoped snapshot is recorded and its best-progress score is reset. Upon entering a new stage, the program initializes the entry-scoped fields in Ī¾zt+1 _z_t+1 from xt+1x_t+1. Stage transitions are irreversible, and final-stage completion terminates the rollout. This same stage structure persists through control and learning, ztz_t conditions the Stage-BC and residual controllers, while the stored snapshot Ī¾zt _z_t preserve the references required by stage-relative relations. For drawer opening, the monitor remains in the drawer-opening stage until all associated predicates, such as ādirection_moveā and āfix_orientationā, are satisfied. Transitioning to the subsequent stage records a new entry snapshot, including the handle and the grippers position and orientation. I-B Program Validation and Acceptance Independent task success. Because the same stage monitor is retained during learning and data generation, its completion logic must be checked against an independent criterion. For each task, we introduce an external evaluator gextg_ext, manually authored independently of SUN Program formation and frozen prior to synthesis. Crucially, gextg_ext is never exposed to the LLM, VLM, SUN Program, or the learned policy. Its binary outcome provides the sparse terminal signal used for residual learning, acceptance, calibration, and final evaluation. In parallel, the SUN Program independently declares completion through its internal predicate DKD_K. A false completion occurs when the SUN monitor declares the final stage complete while gextg_ext reports that the required physical outcome has not been achieved. This separation supplies the independent acceptance criterion used during rollouts. For the same drawer opening task, The independent evaluator gextg_ext only declares success when the drawer joint reaches 12 cm displacement. If SUN monitor declares its final geometric predicates satisfied without gextg_ext , the rollout will be treated as a false completion. MPC screening. Following initial formulation, task-space nonlinear MPC is the first controller to consume the compiled interfaces of Ī , and screens each SUN Program in closed-loop simulation (Step 1 in Figure 2). For stage k, the compiled MPC terms form the stage cost ākā(x~)=āj=1mkĻk,jmpcā(x~) _k( x)= _j=1^m_kĻ^mpc_k,j( x). At state xtx_t, we solve via direct multiple shooting minx1:H,u0:Hā1āh=0Hā1āzt(x~h) _x_1:H,\,u_0:H-1\; _h=0^H-1 _z_t( x_h) (3) s.t.x0=xt,xh+1=fā”(xh,uh). .t. 10.00002ptx_0=x_t, 20.00003ptx_h+1=f(x_h,u_h). Here, H is the planning horizon, f is the prediction model. Within the prediction horizon, x~h=(xh,zt,Ī¾zt) x_h=(x_h,z_t, _z_t), so the active stage and its entry snapshot remain fixed while the problem is solved. Only the first optimized action is executed before re-observation and replanning. After observing the resulting state, Kuafu evaluates the current-stage predicates and gextg_ext before updating the stage index; in the final stage, the predicates determine DKD_K. The MPC problem is then re-solved from the updated runtime state. Acceptance and repair. Each candidate SUN Program is screened under 10 task-sampled initial conditions in closed-loop simulation. The program is accepted when at least one rollout achieves the required physical outcome and no rollout produces a false completion. Failures may trigger up to five whole-program repair attempts based on unsatisfied terms and their measured violations. After acceptance, Kuafu fixes the grounded relations, satisfaction tolerances, predicates, term definitions, and stage order of Ī . The accepted program and its successful MPC trajectories form the handoff to policy learning: the trajectories supervise the Stage-BC policy ĻĪø _Īø, while Ī is retained as the task-semantic core. I-C SUN-Guided Policy Amortization The accepted SUN Program and successful MPC trajectories provide complementary supervision for policy learning. Stage-conditioned behavioral cloning. For each task, we collect 1,000 physically successful MPC trajectories and use them to train a deterministic Stage-BC policy ĻĪø _Īø (Step 2 of Figure 2). The policy maps the current observation and active stage to a 10-step action chunk, and we optimize it using normalized mean-squared error. During learned execution, a reactive policy ĻĪø _Īø replaces the MPC optimization loop at each control step, thereby amortizing the planning computation [3]. The observation oto_t contains the end-effector, gripper, and object/frame poses. Additionally, the policy is conditioned on the active stage ztz_t. The policy predicts 10-step chunks of absolute actions. Training chunks are strictly stage-bound: padding repeats the final action, and only the first predicted action is executed by the residual controller before re-evaluation. Consequently, in the drawer opening example, while securing the grip on the drawer handle, stage-BC chunks cannot cross into the drawer opening manipulation. We use ĻĪø(1)ā(ot,zt) _Īø^(1)(o_t,z_t) to denote the first action in the predicted chunk. Only this action enters the combined policy before the SUN monitor reevaluates the active stage. The retained SUN Program continues to supply ztz_t and carries the task semantics into residual learning. Bounded residual reinforcement learning. We freeze ĻĪø _Īø and use it as the behavior anchor for Step 3, training a bounded additive residual policy ĻĻ _Ļ via PPO with generalized advantage estimation. Following RRL [10], the combined whole-task policy is Ļā”(ot,zt) Ļ(o_t,z_t) =clipā”(ĻĪø(1)ā(ot,zt)+Ī±ĻāĻĻā(ot,zt),umin,umax), =clip\! ( _Īø^(1)(o_t,z_t)+ _Ļ _Ļ(o_t,z_t),u_ ,u_ ), (4) ut u_t =Ļ(ot,zt),āĻĻ(ot,zt)āāā¤Ļmax. =Ļ(o_t,z_t), 20.00003pt \| _Ļ(o_t,z_t) \|_āā¤ _ . Here, Īø and Ļ are the parameters of the Stage-BC and residual policies, respectively. The scalar Ī±Ļ _Ļ controls the residual contribution, Ļmax _ bounds it coordinate-wise, and uminu_ and umaxu_ are the final environment action limits. Program-derived residual reward. For stage k, the same learning costs and weights attached to the terms in Īk _k define the normalized stage score Skā(x~)=1āāj=1mkwk,jāck,jrlā(x~)āj=1mkwk,j,S_k( x)=1- _j=1^m_kw_k,jc^rl_k,j( x) _j=1^m_kw_k,j, (5) where wk,jā„0w_k,jā„ 0, āj=1mkwk,j>0 _j=1^m_kw_k,j>0, and a larger score indicates greater progress within the stage. Kuafu rewards monotonic progress: rt= r_t= Ī»prog[Szt(x^t+1)āSbestzt,tā1]++Ī»stage[zt+1>zt] _prog [S_z_t( x_t+1)-S^best_z_t,t-1 ]_++ _stage 1[z_t+1>z_t] (6) +Ī»trueā[TSt]āĪ»falseā[FCt]āĪ»actāāutā22. + _true 1[TS_t]- _false 1[FC_t]- _act\|u_t\|_2^2. The coefficients Ī»prog _prog, Ī»stage _stage, Ī»true _true, Ī»false _false, and Ī»act _act weight the corresponding reward terms. SbestS^best initializes at zero at episode/stage entry, updates as Sk,tbest=maxā”(Sk,tā1best,Skā(x^t+1))S^best_k,t= (S^best_k,t-1,S_k( x_t+1)), and resets after a stage transition. TStTS_t and FCtFC_t denote the physical success and false-completion events, respectively. An episode terminates at the first such event or at timeout. The remaining terms provide program-derived supervision while the frozen prior and residual bound constrain adaptation. Learning parameters calibration. Inspired by iterative reward refinement [13, 14], Kuafu actively monitors learning trends and constraint satisfaction via SUN Program performance during Step 3 (Figure 2). A liberal residual policy may perform well on short tasks but risks deviation on prolonged tasks demanding precision. These statistics are exposed to an LLM through a constrained schema. We introduce an LLM agent to tune the relative reward weights declared in the accepted program, the residual strength Ī±Ļ _Ļ, and the residual bound Ļmax _ . Crucially, predicate tolerances, logic, grounded relations, stage order, and term definitions remain fixed at their MPC-accepted values. These updates calibrate the learning signal and the permitted deviation from the Stage-BC policy while preserving the task semantics validated by MPC. For example, while pulling the drawer opening, calibration may realize the āfix_orientationā term is easier to reach, and lower its weight relative to ādirection_moveā. I-D Data Production The same semantically fixed program remains active after residual learning. In Step 4, Kuafu supplies ztz_t to the state-based BC and residual controllers, providing stage progression, completion monitoring, and false-completion detection during data generation. For the downstream visual-policy study, trajectories are retained using the common physical-success criterion gextg_ext, ensuring consistent outcome evaluation across all data generators. At the final handoff, the retained trajectories supervise a visual policy using only their recorded observations and actions. The visual policy receives neither ztz_t nor any other output from the SUN monitor during training or deployment. Thus, the accepted SUN Program preserves the correspondence among control, monitoring, and learning throughout data production, while the resulting visual policy executes without the program or privileged state information. IV Experiments Fig. 3: Nine-task evaluation suite in Isaac Lab. TABLE I: Formation, validation, and language interaction cost Metric / method T0 T1 T2 T3 T4 T5 T6 T7 T8 Mean A. SUN Program formation and refinement: canonical accepted programs and their formation cost Stages K 3 8 6 6 7 9 15 14 10 8.67 Constraints 12 25 19 19 27 31 53 44 31 29.00 Formation iterations 1 1 1 1 2 2 2 1 1 1.33 Refinement iterations 0 0 0 0 1 1 0 0 2 0.44 LLM calls/task 11.2 19.8 15.2 16.2 24.6 29.2 37.0 32.8 28.8 23.87 B. Five repeated formation-and-MPC-validation runs per task Success w/o recovery 5/5 5/5 5/5 4/5 0/5 1/5 5/5 2/5 2/5 29/45 Recovery entry 0/5 0/5 0/5 1/5 5/5 4/5 0/5 3/5 3/5 16/45 Final success 5/5 5/5 5/5 5/5 5/5 5/5 5/5 3/5 5/5 43/45 Recovery cycles 0 0 0 2 5 9 0 11 3 30 C. Comparator LLM interaction cost (Kuafu/Eureka: one-time per task; VoxPoser: per rollout) Kuafu tokens/task (k) 37.6 84.0 57.1 62.6 110.1 144.1 183.7 152.5 163.1 110.50 Eureka tokens/task (k) 224.3 246.2 241.1 227.2 208.7 243.0 208.9 248.8 261.0 234.40 VoxPoser calls/run 12.9 13.9 13.0 13.0 16.1 21.8 18.8 19.9 23.5 16.99 VoxPoser tokens/run (k) 25.1 24.6 24.9 23.6 27.2 38.0 31.4 39.7 49.8 31.60 We address four central questions in our experiments: (1) Formulation Reliability: How reliably can SUN Programs be synthesized from language and scene semantics? (2) Performance Elevation: Does the persistent support of SUN Programs significantly elevate task success compared to non-persistent baselines? (3) Data Efficiency: Can Kuafu outperform existing data-generation methods in terms of trajectory quality and sample efficiency? (4) Sim-to-Real Transfer: Does the dataset produced by Kuafu enable visual policies and robust zero-shot transfer to physical robots? As shown in Figure 3, we designed nine multi-stage tasks to evaluate physical capabilities that recur throughout everyday manipulation. The suite spans constrained translational and rotational motionāincluding pushing, pulling, twisting, and reorientingātogether with precise picking and placing, orientation-sensitive grasping, and high-precision manipulation. These capabilities are instantiated through: E-stop interactions (T0: push, T3: twist), stacking (T1: two cubes, T6: three cubes), drawer manipulation (T2: pull open, T7: open and place object), object reorientation (T4: right a bottle), insertion (T5: marker into cup), and dexterous hanging (T8: hang cup by handle). The tasks isolate different forms of compositional scalability for objects and interactions. T0/T3 vary the complexity of contact-constrained motion on the same mechanism. T6 extends T1 by repeating its precision pick-and-place and alignment sequence. T7 combines T2ās constrained drawer motion with the pick-and-place skill exercised in T1.T 4/T5 both require object reorientation, but T5 adds an inter-object geometric constraint by aligning the reoriented marker with the cup opening. Lastly, T5 and T8 compare high-precision manipulation across distinct object morphologies and contact geometries. We report overall performance as task-macro external-outcome success: the unweighted mean of the gextg_ext success rate across all tasks, ensuring each task contributes equally regardless of rollout count. All LLM calls (Kuafu, Eureka, VoxPoser) use GPT-5.5, and simulations run in Isaac Lab 5.1. State-controller evaluations use 8192 parallel environments with rollouts capped at 180 seconds. All methods share the same task-specific initial-condition sampler. IV-A Formulation Reliability Rows A and B of Table I characterize accepted programs and the formation-validation outcomes. Programs span 3ā15 stages and 12ā53 constraints, with each canonical program obtained within one or two formation rounds. A formation round regenerates the program, whereas recovery repairs individual terms flagged by failed MPC screening. Of 45 independent runs, 29 pass initially; diagnostic repair accepted 14 of the remaining 16, raising the yield from 64.4% to 95.6%. Recovery was most critical for T4 and T5, improving acceptance from 1/10 to 10/10. T7 was the only task with unrecovered runs (3/5), accounting for 11 of the 30 total recovery cycles. Since the larger 15-stage T6 program succeeded in all five runs without recovery, raw program size alone does not explain formation difficulty. Instead, the T7 result suggests that combining heterogeneous manipulation phases poses a greater challenge. Within the tested scope, these results establish diagnostic-guided repair as an effective mechanism for recovering initially invalid programs. Rows C report the language interaction. Kuafu uses 110.5k tokens per task, compared with 234.4k for Eurekaās one-time reward search. VoxPoser instead uses 16.99 calls and 31.6k tokens per rollout. At the reported mean rates, its cumulative interaction exceeds Kuafuās after just two executions (calls) and four rollouts (tokens). Thus, forming a persistent program converts language interaction from a recurring execution cost into a task-level cost. IV-B Performance and Ablation Analysis TABLE I: Control-to-learning handoff and baselines Variant / method T0 T1 T2 T3 T4 T5 T6 T7 T8 Mean A. Accepted-program transfer success (%): MPC, Stage-BC, and bounded residual learning (with/without program supervision) MPC 100.00 96.15 100.00 89.29 62.50 58.14 94.34 70.92 82.50 83.76 Stage-BC 36.01 10.79 25.11 29.80 22.47 31.82 13.81 38.80 14.16 24.75 Bounded RRL + Sparse Rewards 58.67 87.20 49.56 58.67 58.96 0.00 0.00 7.95 0.00 35.67 Kuafu (L42L_42) 98.48 95.78 98.01 84.62 58.80 69.70 87.25 95.21 50.43 82.03 Kuafu (L42,L43,L44L_42,L_43,L_44) 98.37Ā± 0.81 94.75Ā± 0.62 97.58Ā± 1.04 82.09Ā± 0.65 56.24Ā± 1.81 62.04Ā± 5.34 84.72Ā± 1.32 94.06Ā± 1.06 44.99Ā± 1.84 79.43 B. False completion rates (%) for Kuafu final controllers Kuafu (L42L_42) 0.37 0.00 0.05 3.26 0.45 0.00 0.00 0.00 1.05 0.58 Kuafu (L42,L43,L44L_42,L_43,L_44) 0.59Ā± 0.57 0.01Ā± 0.00 0.03Ā± 0.02 3.21Ā± 0.27 0.48Ā± 0.07 0.00Ā± 0.00 0.00Ā± 0.00 0.00Ā± 0.00 1.07Ā± 0.50 0.60Ā± 1.05 C. Kuafu component-ablation success (%): partial handoffs and alternative state-policy learners Stage-free-BC 52.20 12.16 3.38 11.17 5.29 0.00 0.00 0.00 0.77 9.44 Pure RL, naive reward 0.04 0.00 0.13 0.00 0.43 0.00 0.00 0.00 0.00 0.07 Pure RL, SUN Program reward 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Stage-free-BC + RRL 1.20 67.30 0.03 0.00 60.31 17.49 3.83 0.16 16.69 18.56 TD-MPC2 [6], naive reward 79.64 27.00 0.00 5.99 1.07 0.00 0.00 0.00 0.00 12.63 TD-MPC2 + prior, naive reward 77.58 13.00 97.47 99.27 0.20 0.00 0.00 0.00 0.00 31.95 D. Baseline performance success (%): iterative reward search (Eureka) and online planning (VoxPoser) Eureka: Train@1 45.14 0.02 20.25 0.19 1.03 0.01 0.00 0.00 0.01 7.41 Eureka: Train@5 0.21 0.01 78.96 8.53 5.29 0.00 0.00 0.00 0.00 10.33 Eureka: Best Train ā¤5ā¤ 5 60.53 0.02 78.96 18.17 6.47 0.01 0.00 0.01 0.01 18.24 Eureka: Final Eval. 25.85 0.01 71.75 15.16 2.83 0.01 0.00 0.00 0.01 12.85 VoxPoser: Direct 50.00 10.00 0.00 20.00 0.00 0.00 10.00 0.00 0.00 10.00 Table IIreports structural handoff controls that remove program-derived progress supervision, the behavioral prior, or the residual constraint. For scalar refinement, Kuafu permits at most 5 residual-RL iterations per task. An iteration is deemed collapsed if its final success falls below 50% of its peak. Refinement halts at the first non-collapsed configuration or upon exhausting the 5-iteration budget, freezing the controller for evaluation. The closest matched ablation isolates program-derived supervision. Starting from the same Stage-BC prior and bounded residual learner, the sparse-reward variant achieves 35.67% task-macro success, versus 82.03% for Kuafuās L42L_42 mean, a 46.36-point gain. Separately, the 3-run robustness aggregate reaches 79.43% success. Because the accepted program, MPC mentor set, and residual training vary jointly across these runs, this result confirms the complete pipelineās effectiveness beyond a single lineage. False-completion rates remain low post-handoff, at 0.60% across the three lineages. The remaining partial handoffs establish complementarity. Stage-BC alone achieves 24.75%. With the SUN Program reward, residual architecture, and action bound held fixed, removing the accepted Stage-BC prior drops performance to 0% (Pure RL, SUN Program reward). Applying residual RL to a stage-free BC reaches only 18.56%. Similarly, adding a behavioral prior raises TD-MPC2 from 12.63% to 31.95%, though success remains concentrated on T0, T2, and T3. Removing or broadening the residual bound collapses all tasks to 0% success; we have omitted the resulting all-zero row from Table I. Neither accepted behavior nor program supervision alone reproduces the complete handoff. Conversely, baselines that rely on iterative reward search or online planning without persistent task semantics struggle to achieve robust, multi-stage success. Eureka exhibits non-monotonic instability: its best candidate within 5 search rounds reaches 18.24% success, but the final policy degrades to 12.85% (e.g., Task T0 drops from 45.14% to 0.21%), indicating reward overfitting. VoxPoser achieves only 10.00% success, failing completely on five tasks. These results confirm that Kuafuās gains stem not from increased LLM search budget, but from the synergistic integration of a verified prior, program-derived supervision, and bounded correction, which eliminate semantic drift. The paired tasks provide a structured test of horizon robustness. Kuafu drops only 8.53 points as stacking grows from 8 stages (T1, 95.78%) to 15 (T6, 87.25%), 2.80 points when drawer opening (T2, 98.01%) is extended to the 14-stage open-and-place task (T7, 95.21%), and 13.86 points from the close-E-stop task (T0, 98.48%) to the more difficult open-E-stop task (T3, 84.62%). Across the three pairs, Kuafu retains 91.4% of its shorter-task performance; the matched sparse-reward handoff instead falls from 65.14% to 22.21%. In a boarder view, no non-Kuafu learned handoff or baseline reaches 50% on either T6 or T7, with taskwise maxima of only 13.81% and 38.80%, whereas Kuafu reaches 87.25% and 95.21%. This shows that the retained SUN Program preserves competence as stage sequences lengthen rather than merely improving isolated tasks. Table IIIdetails the end-to-end compute costs, encompassing program formation, MPC validation, mentor trajectory collection (1000 per task), Stage-BC, and bounded residual learning. In constrast, the Eureka baseline accounts for 5 rounds of 4-candidate reward-searches plus finalization. Under one-GPU worker-hour accounting, Eureka uses 479.60 GPU-hours, compared with 122.67 for Kuafu (a 3.91Ć efficiency gain). Notably, while MPC validation and mentor collection dominate Kuafuās cost, program formation is negligible (1.55 GPU-hours). Thus, Kuafuās superior performance is achieved at a fraction of the computational cost, ruling out compute budget as a confounding factor. TABLE I: End-to-end controller compute (GPU-hrs) Kuafu Eureka Form. MPC BC RRL Total Search Final Total Sum 1.55 69.03 3.21 48.86 122.67 412.24 67.36 479.60 IV-C Data Efficiency TABLE IV: Kuafu data generation A. Downstream DP3 success (%) (500 successful trajectories per task) Task MimicGen SkillMimic Isaac Lab Mimic Isaac Skill Mimic Kuafu T0 94.52 97.07 90.78 72.82 95.70 T1 4.95 3.91 14.21 1.04 58.70 T2 36.12 30.65 40.92 22.54 55.50 T3 33.33 62.00 36.54 15.25 85.30 T4 8.27 8.05 9.35 5.47 0.78 T5 0.00 0.00 0.00 0.00 15.80 T6 0.00 0.00 0.00 0.00 12.70 T7 0.00 0.00 0.00 0.00 69.00 T8 0.00 0.26 0.00 0.26 20.70 Mean 19.69 22.44 21.31 13.04 46.02 B. MPC-relative Q95 yield and successful-trajectory diagnostics Method Q95 (%ā ) Time/ MPC50MPC_50 ā EEF path/ MPC50MPC_50 ā Action path/ MPC50MPC_50 ā Action Ī2 ^2/ MPC50MPC_50 ā MimicGen 20.4 1.11 1.14 1.94 1.27 SkillMimic 6.9 1.66 1.15 3.01 1.28 Isaac Lab Mimic 12.1 1.12 1.38 1.96 1.25 Isaac SkillMimic 6.2 1.61 1.46 2.89 1.21 Kuafu (ours) 57.3 1.04 0.67 0.52 0.28 Human reference 0.0 3.06 1.59 2.81 1.25 C. Successful trajectory-time throughput (min/active production hr, ā ) Task Human MimicGen SkillMimic Isaac Lab Mimic Isaac Skill Mimic Kuafu Task macro 23.34 29.36 26.48 32.19 24.79 246.79 vs. Human 1.00 1.26 1.13 1.38 1.06 10.57 D. Final-controller readiness and successful-demonstration break-even Task Controller ready (GPU h) Human demos at ready Break-even demos Post-ready collect (min) Task macro 13.63 494 513 29.10 Across-task range 4.84ā25.52 222ā805 225ā854 4.35ā76.19 TABLE V: Visual policy performance and zero-shot transfer A. Single-task success (%) by visual-policy architecture (same Kuafu data) RGB policy Mean Point-cloud policy Mean ACT [31] 35.40 DP3 46.02 Diffusion Policy 38.52 FlowPolicy [30] 21.01 smolVLA [21] 26.73 ManiFlow-PC [27] 46.46 ManiFlow-RGB [27] 34.09 MP1 [20] 12.94 B. Per-task modality maxima (RGB vs. point-cloud) and state-controller success (%) T0 T1 T2 T3 T4 T5 T6 T7 T8 Mean RGB max 84.53 30.54 73.98 60.70 25.50 10.10 12.10 73.00 23.00 43.72 Point-cloud max 98.38 65.53 86.28 85.30 10.05 15.80 12.70 69.00 20.70 51.53 State controller 98.48 95.78 98.01 84.62 58.80 69.70 87.25 95.21 50.43 82.03 C. Multi-task VLA success (%) (single policy across T0āT8) Policy T0 T1 T2 T3 T4 T5 T6 T7 T8 Mean SmolVLA [21] 5.87 0.26 12.94 2.08 0.00 0.00 0.00 0.00 0.00 2.35 WALL-X [29] 22.22 1.56 60.07 11.78 0.00 0.00 0.00 10.51 0.00 11.79 Ļ0.5 _0.5 [1] 26.61 0.0 81.87 0.00 4.17 4.17 0.00 69.17 0.00 20.67 D. Zero-shot Sim2Real (DP3) transfer to physical robots Franka FR3 Franka Hand Franka FR3 Robotiq 2F-85 Kinova Gen3 Robotiq 2F-85 Task T1 T2 T7 T8 T4 T5 Task Macro Trials 20 20 20 30 10 6 Success 3 11 7 9 4 2 Rate (%) 15.0 55.0 35.0 30.0 40.0 33.3 34.72 During data collection, the learned controller replaces online MPC, while the retained SUN Program continues to supply stage annotations, completion monitoring, diagnostics, and false-completion filtering. Table IV evaluates downstream utility under a matched successful-data budget, while also reporting MPC-relative trajectory diagnostics, steady-state production throughput, and controller amortization. To assess efficiency and smoothness beyond physical success, we define a rollout as Q95-qualified only if it satisfies gextg_ext and its completion time, end-effector path length, action path length, and action second difference are each no greater than the corresponding task-specific 95th percentile among successful MPC trajectories. Q95 yield is the fraction of evaluated rollouts that satisfy all of these conditions. As an MPC-relative metric, Q95 measures mentor-aligned efficiency and smoothness rather than universal quality; thus, the fixed-learner test reported in Rows A of Table IV serves as the primary utility measure. For reporting, metrics (time, paths, action derivatives) are normalized by task-specific successful-MPC medians (MPC50). Human baselines were collected via simulation teleoperation using an AgileX PIKA gripper, with measured poses defining the simulated end-effector targets. The simulated manipulator is still subjected to dynamics constraints and collisions. Rows A of Table IV provide the primary data-utility comparison. Under a matched budget of 500 successful trajectories per task and a fixed DP3 learner, Kuafu data reach 46.02% task-macro success, compared with 22.44% for SkillMimic, the strongest alternative, a 2.05Ć gap. The seed-42 source leads on seven tasks and alone gives nonzero downstream success on all nine. Since L42L_42 is used exclusively for this comparison (excluding L43L_43 and L44L_44), the result isolates data quality from selection bias. On T5āT7, all competing sources achieve 0%, whereas Kuafu reaches 15.80%, 12.70%, and 69.00%, respectively. Thus, Kuafuās advantage stems from downstream utility, not data volume. Rows B of Table IV characterize trajectory fidelity relative to the accepted MPC reference. Kuafu attains a 57.3% Q95 yield (vs. 20.4% for MimicGen, a 2.81Ć gain). Among successful rollouts, it produces the shortest paths and smoothest actions (metrics: 1.04 completion time, 0.67 end-effector path, 0.52 action path, 0.28 action second difference). Rows C and Rows D of Table IV separate steady-state data-generation throughput from the one-time generator cost. With the final state controller, Kuafu generates 246.79 successful trajectory minutes per GPU-hour, a 10.57Ć increase over human teleoperation (23.34 min/hr). IV-D Visual Learning and Zero-Shot Transfer Table Vpresents the utility of Kuafuās trajectories after removing privileged state and stage inputs. Across four RGB and four point-cloud architectures, every single-task policy family achieves nonzero task-macro success. Notably, ManiFlow-PC reaches 46.46% and DP3 reaches 46.02%, outperforming the strongest RGB policy 38.52%. Rows B report per-task maxima within each modality family; these represent optimal architectures for each task, not a single deployable oracle. While maxima nearly match the state controller on T0 (98.38% vs. 99.61%) and exceed it on T3 (85.30% vs. 84.95%), the 15-stage T6 shows the largest gap (12.70% vs. 87.51%). These gaps, along with the multi-task results, reflect losses from observation conditioning, partial observability, and architectural constraints rather than state-based execution alone. For physical deployment, each simulated setup reproduces the calibrated two or three RGBD-cameras configuration of the coresponding workspace, with constrained randomization. We double the dataset to 1,000 trajectories per task and deploy the resulting policies without real-world fine-tuning. DP3 achieves 36 successes in 106 trials across six tasks and three robot-gripper configurations without real-world fine-tuning. This yields a 34.72% task-macro and nonzero success on every tested task. The pooled rate is 33.96% (95% Wilson interval: 25.6ā43.4%, with the smallest task showing 2/6 successes (9.7%ā70.0%). Inspection of failed trials identifies precise grasp acquisition as the main remaining challenge: T1 has the lowest success rate at 15%, and grasping is the most frequent observed failure mode on T7. These results establish zero-shot sim-to-real feasibility using dataset produced under persistent SUN programs. V Conclusions We presented Kuafu, a control-to-learning framework that preserves an MPC-screened Semantically UNified (SUN) Program as a persistent, executable task artifact alongside a learned policy. Across three independent lineages, Kuafu achieves an average 79.43% task-macro success rate, demonstrating end-to-end robustness on multi-stage manipulation tasks. In our evaluation, the complete handoff reaches 82.03% success, significantly outperforming the sparse-reward baseline (35.67%) and alternative learners; ablation studies further isolate the distinct contributions of the verified behavioral prior and bounded residual correction. Leveraging the SUN Programās supervision in data generation, Kuafu trajectories yield 46.02% success in fixed-volume DP3 training, support eight single-task visual architectures and three multi-task VLAs, and enable zero-shot transfer across 106 physical trials. Results establish a key principle: task semantics validated through control remain effective even after behavior is amortized into a learned policy. Scope and limitations. SUN Programs rely on a registered scene interface with informative object frames and axes, as well as a finite, typed-operator library. Consequently, Kuafu cannot handle deformable objects or fluids without defining new operators. Furthermore, our evaluation does not cover held-out instructions, operators, scene semantics, or task compositions, limiting strong claims of open-world generalization. Methodologically, the stage monitor advances monotonically and cannot rollback after a physical regression. Additionally, the Q95 metric is MPC-relative, measuring efficiency against a specific planner rather than absolute trajectory quality. Finally, the validation (6 task, 106 trials) establishes zero-shot feasibility across the tested robot-gripper configurations but does not guarantee broad sim-to-real reliability in unstructured environments. References [1] K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. R. Equi, C. Finn, N. Fusai, et al. (2025) Ļ0.5 _0.5: A vision-language-action model with open-world generalization. In Proc. Conf. Robot Learn. (CoRL), Vol. 305, p. 17ā40. Cited by: TABLE V. [2] S. Booth, W. B. Knox, J. Shah, S. Niekum, P. Stone, and A. Allievi (2023) The perils of trial-and-error reward design: misdesign through overfitting and invalid task specifications. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37, p. 5920ā5929. External Links: Document Cited by: Ā§I. [3] A. Byravan et al. (2022) Evaluating model-based planning and planner amortization for continuous control. In Proc. Int. Conf. Learn. Represent. (ICLR), Cited by: Ā§I-C. [4] J. Carius, F. Farshidian, and M. Hutter (2020) MPC-Net: a first principles guided policy search. IEEE Robotics and Automation Letters 5 (2), p. 2897ā2904. External Links: Document Cited by: Ā§I, Ā§I, Ā§I. [5] C. R. Garrett, A. Mandlekar, B. Wen, and D. Fox (2025) SkillMimicGen: automated demonstration generation for efficient skill learning and deployment. In Proc. Conf. Robot Learn. (CoRL), Vol. 270, p. 2750ā2790. External Links: Link Cited by: Ā§I. [6] N. Hansen, H. Su, and X. Wang (2024) TD-MPC2: scalable, robust world models for continuous control. In Proc. Int. Conf. Learn. Representations (ICLR), Cited by: TABLE I. [7] P. Hua, M. Liu, A. Macaluso, Y. Lin, W. Zhang, H. Xu, and L. Wang (2025) GenSim2: scaling robot data generation with multi-modal and reasoning llms. In Proc. Conf. Robot Learn. (CoRL), Vol. 270, p. 5030ā5066. External Links: Link Cited by: Ā§I. [8] W. Huang, C. Wang, Y. Li, R. Zhang, and L. Fei-Fei (2025) ReKep: spatio-temporal reasoning of relational keypoint constraints for robotic manipulation. In Proc. Conf. Robot Learn. (CoRL), Vol. 270, p. 4573ā4602. External Links: Link Cited by: Ā§I, Ā§I. [9] W. Huang, C. Wang, R. Zhang, Y. Li, J. Wu, and L. Fei-Fei (2023) VoxPoser: composable 3D value maps for robotic manipulation with language models. In Proc. Conf. Robot Learn. (CoRL), Vol. 229, p. 540ā562. External Links: Link Cited by: Ā§I. [10] T. Johannink, S. Bahl, A. Nair, J. Luo, A. Kumar, M. Loskyll, J. A. Ojea, E. Solowjow, and S. Levine (2019) Residual reinforcement learning for robot control. In Proc. IEEE Int. Conf. Robot. Autom. (ICRA), p. 6023ā6029. External Links: Link Cited by: Ā§I, Ā§I-C. [11] G. Kahn, T. Zhang, S. Levine, and P. Abbeel (2017) PLATO: policy learning using adaptive trajectory optimization. In 2017 IEEE International Conference on Robotics and Automation, p. 3342ā3349. External Links: Document Cited by: Ā§I, Ā§I. [12] S. Levine and V. Koltun (2013) Guided policy search. In Proc. Int. Conf. Mach. Learn. (ICML), Vol. 28, p. 1ā9. External Links: Link Cited by: Ā§I. [13] Y. J. Ma, W. Liang, G. Wang, D. Huang, O. Bastani, D. Jayaraman, Y. Zhu, L. Fan, and A. Anandkumar (2024) Eureka: human-level reward design via coding large language models. In Proc. Int. Conf. Learn. Representations (ICLR), External Links: Link Cited by: Ā§I, Ā§I, Ā§I-C. [14] Y. J. Ma, W. Liang, H. Wang, S. Wang, Y. Zhu, L. Fan, O. Bastani, and D. Jayaraman (2024) DrEureka: language model guided sim-to-real transfer. In Robotics: Science and Systems, External Links: Link Cited by: Ā§I, Ā§I-C. [15] A. Mandlekar, S. Nasiriany, B. Wen, I. Akinola, Y. Narang, L. Fan, Y. Zhu, and D. Fox (2023) MimicGen: a data generation system for scalable robot learning using human demonstrations. In Proc. Conf. Robot Learn. (CoRL), External Links: Link Cited by: Ā§I. [16] D. Q. Mayne, J. B. Rawlings, C. V. Rao, and P. O. M. Scokaert (2000) Constrained model predictive control: stability and optimality. Automatica 36 (6), p. 789ā814. External Links: Document Cited by: Ā§I. [17] Y. Meng and C. Fan (2023) Signal temporal logic neural predictive control. IEEE Robotics and Automation Letters 8 (11), p. 7719ā7726. External Links: Document, Link Cited by: Ā§I. [18] T. Mu, M. Liu, and H. Su (2024) DrS: learning reusable dense rewards for multi-stage tasks. In Proc. Int. Conf. Learn. Representations (ICLR), External Links: Link Cited by: Ā§I. [19] M. Pan, J. Zhang, T. Wu, Y. Zhao, W. Gao, and H. Dong (2025) OmniManip: towards general robotic manipulation via object-centric interaction primitives as spatial constraints. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), p. 17359ā17369. External Links: Link Cited by: Ā§I. [20] J. Sheng, Z. Wang, P. Li, and M. Liu (2026) MP1: MeanFlow tames policy learning in 1-step for robotic manipulation. In Proc. AAAI, Vol. 40. External Links: Document Cited by: TABLE V. [21] M. Shukor, D. Aubakirova, F. Capuano, et al. (2025) SmolVLA: a vision-language-action model for affordable and efficient robotics. arXiv:2506.01844. Cited by: TABLE V, TABLE V. [22] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg (2023) ProgPrompt: generating situated robot task plans using large language models. In Proc. IEEE Int. Conf. Robot. Autom. (ICRA), p. 11523ā11530. External Links: Link Cited by: Ā§I. [23] R. Toro Icarte, T. Q. Klassen, R. Valenzano, and S. A. McIlraith (2018) Using reward machines for high-level task specification and decomposition in reinforcement learning. In Proc. Int. Conf. Mach. Learn. (ICML), Vol. 80, p. 2107ā2116. External Links: Link Cited by: Ā§I. [24] M. Toussaint (2015) Logic-geometric programming: an optimization-based approach to combined task and motion planning. In Proc. Int. Joint Conf. Artif. Intell. (IJCAI), p. 1930ā1936. External Links: Link Cited by: Ā§I, Ā§I. [25] Y. Wang, Z. Xian, F. Chen, T. Wang, Y. Wang, K. Fragkiadaki, Z. Erickson, D. Held, and C. Gan (2024) RoboGen: towards unleashing infinite data for automated robot learning via generative simulation. In Proc. Int. Conf. Mach. Learn. (ICML), Vol. 235, p. 51936ā51983. External Links: Link Cited by: Ā§I. [26] T. Xie, S. Zhao, C. H. Wu, Y. Liu, Q. Luo, V. Zhong, Y. Yang, and T. Yu (2024) Text2Reward: reward shaping with language models for reinforcement learning. In Proc. Int. Conf. Learn. Representations (ICLR), Note: Spotlight External Links: Document, 2309.11489, Link Cited by: Ā§I. [27] G. Yan, J. Zhu, Y. Deng, S. Yang, R. Qiu, X. Cheng, M. Memmel, R. Krishna, A. Goyal, X. Wang, and D. Fox (2025) ManiFlow: a general robot manipulation policy via consistency flow training. In Proc. Conf. Robot Learn. (CoRL), Vol. 305, p. 2268ā2293. Cited by: TABLE V, TABLE V. [28] W. Yu, N. Gileadi, C. Fu, S. Kirmani, K. Lee, M. G. Arenas, H. L. Chiang, T. Erez, L. Hasenclever, J. Humplik, B. Ichter, T. Xiao, P. Xu, A. Zeng, T. Zhang, N. Heess, D. Sadigh, J. Tan, Y. Tassa, and F. Xia (2023) Language to rewards for robotic skill synthesis. In Proc. Conf. Robot Learn. (CoRL), Vol. 229, p. 374ā404. External Links: Link Cited by: Ā§I. [29] A. Zhai, B. Liu, B. Fang, C. Cai, E. Ma, E. Yin, H. Wang, H. Zhou, J. Wang, L. Shi, L. Liang, M. Wang, Q. Wang, R. Gan, R. Yu, S. Li, S. Liu, S. Chen, V. Chen, and Z. Xu (2025) Igniting VLMs toward the embodied space. arXiv preprint arXiv:2509.11766. Cited by: TABLE V. [30] Q. Zhang, Z. Liu, H. Fan, G. Liu, B. Zeng, and S. Liu (2025) FlowPolicy: enabling fast and robust 3D flow-based policy via consistency flow matching for robot manipulation. In Proc. AAAI, Vol. 39, p. 14754ā14762. External Links: Document Cited by: TABLE V. [31] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn (2023) Learning fine-grained bimanual manipulation with low-cost hardware. In Proc. Robot.: Sci. Syst. (RSS), Daegu, Republic of Korea. External Links: Document Cited by: TABLE V. [32] H. Zhu, A. Gupta, A. Rajeswaran, S. Levine, and V. Kumar (2019) Dexterous manipulation with deep reinforcement learning: efficient, general, and low-cost. In 2019 International Conference on Robotics and Automation (ICRA), p. 3651ā3657. External Links: Document Cited by: Ā§I.