Paper deep dive
Make Code as Policy Great Again: Frontier Agents Write, Call, and Evolve Robot Tools
Shijia Ge, Alex Zhou, Jianshu Zeng, Yexing Wan, Di Wu, Zelin Zheng, Yazhe Wang, Zhiqi Jia, Xuan Shangguan, Jay Zhu, Yijun Liu, Lingyu He, Sihang Wu, Xiao He, Hongcheng Gao
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Frontier models can control robots, but reasoning through every reach, grasp, and retreat makes manipulation slow and token-intensive. We revisit code as policy with a different division of labor: models build executable tools, code handles multi-phase motions, and models decide what to do next. We introduce URAI (Universal Robot-Agent Interface), which couples a programming agent that constructs robot tools with an execution agent that uses them in a feedback loop. The programming agent writes reusable and task-specific tools from task intent and refines them through execution feedback and human guidance. The execution agent selects and parameterizes these tools from current observations; each call runs a complete motion locally before returning control to the agent. Unlike delegating subsequent decisions to a generated program, this design retains model-level decision-making between tool executions. Validated tool revisions persist across episodes without updating foundation-model weights, and a shared GUI and API make the same tools available to humans and agents. Across five RoboDojo tasks and four frozen execution agents, URAI raises aggregate success from 18.0% to 53.0% relative to direct fingertip control, with the largest gain on Swap Blocks; with the same tools, a program written in advance reaches only 24% against 56% for two agents deciding after each call. Three of the four agents also finish episodes 1.3-1.5 times faster with 1.5-1.7 times fewer execution-agent output tokens; DeepSeek-V4-Flash's cost barely changes. We further evaluate URAI on seven real-world AgileX dual-arm tasks, spanning object manipulation, cloth folding, and human-interactive tic-tac-toe. URAI connects the coding and decision-making capabilities of frontier agents, organizing robot control around reusable tools that agents can both invoke and revise.
Tags
Links
- Source: https://arxiv.org/abs/2609.39018v1
- Canonical: https://arxiv.org/abs/2609.39018v1
Trouble viewing inline? Open PDF directly →
Full Text
85,370 characters extracted from source content.
Expand or collapse full text
Make Code as Policy Great Again: Frontier Agents Write, Call, and Evolve Robot Tools Shijia Ge 1 Alex Zhou 1 Jianshu Zeng 2 Yexing WanDi Wu 3 Zelin Zheng 1 Yazhe Wang 3 Zhiqi Jia 1 Xuan ShangguanJay Zhu 1 Yijun Liu 4 Lingyu He 1 Sihang Wu 1 Xiao He 1,† Hongcheng Gao 1,4,† 1 Hexafuture Inc. 2 Peking University 3 Beijing Institute of Technology 4 Tsinghua University † Corresponding authors Abstract Frontier models can control robots, but reasoning through every reach, grasp, and retreat makes manipulation slow and token-intensive. We revisit code as policy with a different division of labor: models build executable tools, code handles multi-phase motions, and models decide what to do next. We introduce URAI (Universal Robot–Agent Interface), which couples a programming agent that constructs robot tools with an execution agent that uses them in a feedback loop. The programming agent writes reusable and task-specific tools from task intent and refines them through execution feedback and human guidance. The execution agent selects and parameterizes these tools from current observations; each call runs a complete motion locally before returning control to the agent. Unlike delegating subsequent decisions to a generated program, this design retains model-level decision-making between tool executions. Validated tool revisions persist across episodes without updating foundation-model weights, and a shared GUI and API make the same tools available to humans and agents. Across five RoboDojo tasks and four frozen execution agents, URAI raises aggregate success from 18.0% to 53.0% relative to direct fingertip control, with the largest gain on Swap Blocks; with the same tools, a program written in advance reaches only 24% against 56% for two agents deciding after each call. Three of the four agents also finish episodes 1.3–1.5×faster with 1.5–1.7×fewer execution-agent output tokens; DeepSeek-V4-Flash’s cost barely changes. We further evaluate URAI on seven real-world AgileX dual-arm tasks, spanning object manipulation, cloth folding, and human-interactive tic-tac-toe. URAI connects the coding and decision-making capabilities of frontier agents, organizing robot control around reusable tools that agents can both invoke and revise. 1 Introduction Frontier models such as GPT-6 Astra, which reconstructs three-dimensional objects by generating CAD code [32] and builds Blender scenes through computer use [34], also complete physical manipulation tasks by selecting end-effector targets and gripper commands from visual observations [35]: it inspects the scene, chooses a Cartesian pose and gripper state for the robot’s inverse kinematics and local controller to realize, and observes the outcome before deciding what to do next. In this setting the model itself is the policy, one motion at a time; GPT-Policy studies it on real robots with in-context demonstrations and human interaction [8]. The difficulty is speed. When a manipulation is decomposed into many model-selected motions, each observation– reasoning–action cycle adds waiting time and tokens: in an independent block-placement evaluation, GPT-6 Astra completed 19 of 20 trials at a mean of 2.5 minutes and 2.1k output tokens per run [35], and InSight separates model latency from robot motion in a comparison with CaP-X [41]. Deployment therefore requires measuring how long a 1 arXiv:2609.39018v1 [cs.RO] 30 Sep 2026 (a) Agent in the loop through URAI Code as policy: one tool call runs one complete, timed motion Execution agent GPT-6ClaudeHuman reads the new observation, then picks the next tool call Programming agent GPT-6Claude writes, validates, and freezes tools 1 tool call arm, pixels, ... 3 feedback + new observation failures validated, frozen tools URAI GUI | API Universal Robot– Agent Interface Tools L PnPPushButton...T(τ) 2 Ground pixels 3D Plan timed path Execute local, 50 Hz → Tic-tac-toe vs. a human PnP(...) Unscrew a bottle cap bottle_cap_twist(...) (b) What one model call controls model callrobot motion Direct end-effector control one call per end-effector command the model decides every step many calls: slow and token-heavy Code as Policies the model writes the program first the code decides what runs next handles only what the code checks Code as Policy with URAI one call per complete, timed motion the agent decides each next call agent in the loop between motions Figure 1. URAI overview. (a) A programming agent writes and validates reusable and task-specific tools. An execution agent selects tools from observations and feedback, while a shared GUI/API backend grounds, checks, and executes multi-phase motions locally. Fading outlines in the demonstration-video frames indicate earlier arm poses within one tool call. (b) Decision boundaries: direct control queries the model for each end-effector command; Code as Policies delegates subsequent decisions to generated code, which may query perception; URAI returns control to the agent between tool executions. successful interaction takes, alongside whether it succeeds. Action granularity is one lever. A model asked to specify successive end-effector targets must manage motion details that a local tool can handle; a tool can instead combine several motion phases into one parameterized request, reusable across tasks or written for one. VLA models predict tokenized or continuous low-level actions [24,44] but generalize poorly to new scene layouts: on RoboDojo, VLA and world action model (WAM) policies keep a median of 12–16% of their success when the layouts are randomized, whereas GPT-6 Astra acting through RoboDojo’s own RoboProbe tool harness keeps 87% on the official leaderboard [36]. Agentic interfaces expose discrete motion commands to a frozen VLM [7,14], and program- and constraint-based systems delegate execution to local controllers [18,26]; the design choice is which decisions stay with the model. Software development offers a template: with coding agents, developers state what they want, run what the agent writes, and ask for changes, a practice Karpathy[23]named vibe coding. We ask whether robot control can work the same way: can the policy be code that agents write and revise from intent and feedback, so that a frontier model decides which complete motion to run instead of every step, and does this cut end-to-end time and tokens while keeping manipulation reliable? Code as Policies showed that a language model can write a robot policy as a program, generated hierarchically and able to call perception [26]; at run time its code, not the model, decides what runs next, and multi-turn variants regenerate code from execution feedback between runs [11]. We make code as policy great again with frontier coding agents, keeping the agent in the loop between tool executions (Figure 1). The robot policy combines agent-written executable tools with feedback-driven tool selection by a frozen foundation model, the execution agent; a programming agent vibe-codes the tools from natural-language intent and execution feedback, starting from reusable tools such as pick-and-place (PnP), PushButton, or Home and writing a task-specific tool when these do not adequately cover a task. The execution agent selects and parameterizes a tool from the current observation and decides again from the feedback after every tool execution; a PnP request covers approach, grasp, transfer, placement, and retreat without a model call for each phase. Unlike casual vibe coding, a tool enters the policy only after it passes validation, and the version used for evaluation is frozen; the two roles separate the cost of writing the policy from the cost of running it. Direct end-effector control also leaves inverse kinematics and local control to the robot; what changes is the granularity and frequency of model decisions: one call executes a complete multi-phase motion, and the model decides the next step from feedback between calls. 2 The tools run behind URAI, a universal robot–agent interface that exposes the same tool contract through a GUI and a structured API over one backend that grounds requests, checks planned motion, and executes locally with explicit paths, orientations, timing, and gripper events; humans can steer the programming agent in natural language and operate the same tools through the GUI. Because the tools are code, the policy can also improve without training: guided by failures and natural-language steering, the programming agent revises the tool collectionL, tool codeC, and execution strategy Pwith foundation-model weights fixed; validated versions are retained for later episodes and frozen before evaluation. Recent systems share parts of this design: Agent as Policy drives a real robot with a coding agent that writes and revises programs from physical feedback, with separate preparation and execution agents [21]; CaP-X adds multi-turn feedback and skill synthesis to Code as Policies [11]; ENPIRE and ASPIRE refine code policies from rollouts [30,42]; LATM, CRAFT, and Voyager split tool makers from users or accumulate code skills [5,40,45]. What distinguishes URAI is the handoff boundary: a fixed tool collection, a frozen execution agent that decides again after every complete tool execution, and the same tool contract for humans through a GUI, evaluated against direct low-level control and against a program written in advance over the same tools. Our contributions are fourfold. (1) Method: code as policy with frontier agents in every role. A programming agent writes the robot tools; each tool call executes a timed end-effector path that the backend grounds, checks, and runs locally; and a frozen execution agent orchestrates all of these tools through URAI, a universal robot–agent interface whose GUI and API share one tool contract for humans and agents. (2) Performance, extensibility, and efficiency: without robot training data, Claude Opus 5.5 through URAI reaches a 60.0% five-task mean over 25 episodes on RoboDojo, close to the best learned policy on the public leaderboard (58.1%, under its own protocol); in our experiments, a new task called for writing or revising tools rather than collecting demonstrations and retraining; and for three of the four models, deciding one complete motion at a time is faster and uses fewer tokens than step-by-step control by the same model. (3) Tool evolution in a feedback loop: the execution agent decides again from the feedback of every call, and the programming agent revises the tool collection and tool code across episodes without weight updates, as in the author-guided extension of a rigid-object PnP tool to cloth folding. (4) Simulation and real-robot evidence: on five RoboDojo tasks [6] with four execution agents, URAI raises success from 18.0% to 53.0% over direct fingertip control, deciding after each call reaches 56% against 24% for a program written in advance over the same tools, and three of the four agents finish 1.3–1.5×faster with fewer output tokens; on seven AgileX dual-arm tasks, against published runs of the same model on other robots, URAI finishes tic-tac-toe 2.7×faster than GPT-Policy with the same 3/3 successes and block into bowl 2.4× faster with 4.7× fewer output tokens than direct end-effector control. 2 Related Work Frontier models as robot decision makers.Code as Policies establishes hierarchical program generation, with perception calls inside the program, as an interface to perception and control [26]; CaP-X extends it with multi-turn interaction, structured execution feedback, and automatic skill synthesis, and exposes a trade-off between the convenience of higher- level primitives and the expressivity of lower-level control [11]; Agent as Policy drives a physical robot with a coding agent that writes programs and revises them from physical feedback, with a preparation agent for reusable task definitions and an execution agent per trial [21]. VoxPoser composes three-dimensional value maps [17], MOKA and PIVOT use visual marks and iterative visual prompting [27,31], ReKep generates constraints that a local optimizer converts into reactive motion [18], RoboStream keeps a frozen VLM in a closed loop through an object-centric scene-graph memory [19], and Inner Monologue feeds environment feedback back into language-model planning [16]; GPT-Policy takes the step-by-step route with a frontier VLM on real robots [8]. Persistent improvement outside model weights has precedents in Voyager’s code skill library [40], in LATM and CRAFT, which separate tool makers from tool users for language agents [5,45], and in robotics in ASPIRE’s validated skill repairs [30], ENPIRE’s reset–execute–verify–refine loop on physical rollouts [42], and RHO’s search over policy repositories against simulated reward [10]. URAI is distinguished by the handoff boundary it evaluates: a fixed collection of agent-written tools, a frozen execution agent that decides again after every complete tool execution, and one tool contract shared by the GUI and the API. Frontier-model control and action granularity. Robocurve evaluates GPT-6 Astra on physical YAM arms through absolute end-effector pose commands, with a 20-call budget and a 25% speed cap [35], and Su et al.[37]compare direct control with hybrid control that accepts or correctsπ 0.5 proposals. OpenVLA predicts learned, quantized per- dimension motor tokens [24], Qwen-RobotManip emits continuous action chunks [44], VGGT-DP and Spatial Policy 3 ground learned visuomotor policies in 3D foundation features and spatially conditioned video prediction [12,28], and RoboDawn [14] and Show-Harness [7] give a frozen VLM a fixed vocabulary of discrete translation, rotation, and gripper commands. URAI also presents discrete tool calls, but a call can run a reusable tool such as PnP or a task-specific procedure whose internal motion phases execute locally; the distinction is the duration and responsibility assigned to one model-level decision, together with the cost of authoring the tool. Interfaces and efficiency. Trajectory and sketch interfaces condition learned policies or frontier agents on rough trajectories, image-space paths, editable sketches, sparse waypoints, or a browser-based three-dimensional interface [13,15,25,38,39], and human interfaces range from sketching [20] to teleoperation and demonstration capture [9,46,48]; Show-Harness’s GUI lets humans and VLMs issue the same single-step units [7]. URAI likewise shares one action vocabulary between human and model operators, but each entry is a complete, timed tool with explicit metric arguments rather than a fixed increment. Computer-use work treats the observation/action boundary as part of an agent system [2,43,49] and motivates measuring efficiency in addition to success [1]; in robotics, InSight reduces VLM thinking time through acquired primitives [41], and Jev and RoboJEV shorten each decision [3,29]. These systems lower the cost of each decision; URAI instead lowers the number of decisions per episode. 3 Code as Policy with the Agent in the Loop 3.1 The Policy: Agent-Written Tools and Feedback-Driven Selection We write the robot policy asπ θ,φ , whereθare the fixed weights of the foundation model behind the execution agent and φ = (L,C,P)is the external system: the tool collectionL, the executable tool codeC, and the execution strategyP, meaning the task guidance and call conventions given to the execution agent. At each decision, the execution agent maps the instruction, the current observation, and the execution history to one tool call(ℓ,ξ)withℓ∈ Land arguments ξ. The codeC ℓ turns the call into a timed motion that runs locally until it completes or is interrupted, so one policy decision spans a complete multi-phase motion rather than one end-effector target. Adaptation happens inφ, never inθ. The programming agent proposesφ k+1 fromφ k , development feedback, and natural-language intent, runs the candidate, and keeps it only if it passes validation: a vibe-coding loop [23] in which every accepted version is tested and frozen before evaluation. 3.2 Interaction Contract The programming agent receives a task specification, available tool definitions, and development feedback, and can revise a reusable tool or create a task-specific tool, including its code, argument contract, and execution strategy. The execution agent, or a human calling the tools ofLthrough the GUI with the same execution semantics, receives the task instruction, camera and depth-derived observations, robot state, and execution feedback, and invokes an available tool through the GUI or API. The shared backend grounds and validates each request, produces a timed trajectory, and executes it locally; after completion or interruption the execution agent can observe again and issue another request, and recurrent failures can be passed to the programming agent for a versioned tool revision (Figures 1 and 2). 3.3 A Motion Representation with Explicit Timing For an arm, we describe a request as a = p(s),R(s),v(s),E,C , s∈ [0, 1],(1) wherep(s)∈R 3 is a metric path,R(s)∈ SO(3)is a desired end-effector orientation,v(s)is the requested translational speed profile,Eis an ordered set of gripper events and holds, andCspecifies execution settings such as pose tolerances and orientation constraints. Progresssindexes the path; it is not wall-clock time. The prototype accepts sampled control points and bounded mathematical expressions, and interpolates rotations onSO(3). For a regular translating segment with positive requested speed, the nominal timing relation is dt ds = ∥p ′ (s)∥ 2 v(s) .(2) 4 (a) Tool programming and persistent evolution between episodes: the programming agent revises φ k φ k+1 ; foundation-model weights θ stay fixed → Development episodes task outcome failure cases θ fixed Programming agent GPT-6 Claude writes or fixes a tool contract + code Candidate φ k+1 L Tool collection PnP, PushButton, T(τ) C Tool code executable definition P Execution strategy control and recovery Validate contracts regression cost Freeze φ k+1 held-out tasks fixed budget (b) Closed-loop execution within an episode: one tool per model call, run locally RGB-D input Task “Fold the clothes neatly.” θ fixed Execution agent GPT-6 Claude Human (GUI) selects a tool and fills its arguments GUI | API tool call URAI φ k = (L k , C k , P k ) Tool collection L k PnPPushButtonT(τ) Ground geometry Plan limits Execute C k , P k timed path · orientation · gripper Execution Task evaluator success / failure new observation + execution feedbackrecurrent failures accepted tools Figure 2. Tool development and execution in URAI. (a) Between episodes, the programming agent revises the tool collection, code, and execution strategy using development feedback. Validated versions are frozen before evaluation; model weights remain fixed. (b) Within an episode, the execution agent selects and parameterizes tools from observations and feedback, while the backend executes each motion locally. This is the classical path–velocity decomposition [4,22,33]; what the interface adds is that path, speed profile, and gripper events are the arguments of one tool call. Gripper events are tied to motion progress and execution conditions; ordinary placement and dynamic release use different event semantics. 3.4 Grounding, Editing, and Local Execution Points selected on observed surfaces are back-projected from calibrated depth, whereas the height of a free-space path is edited in depth-derived orthographic side views. The GUI adds continuous drawing, arm selection, orientation handles, and a speed-profile editor; the API exposes the corresponding control points, profiles, and motion settings through a bounded arithmetic interpreter, so an agent can specify an entire curve without issuing a mouse action for each point. The backend checks the full trajectory, including the approach from the current pose, against the configured kinematic and geometric constraints and streams references locally at a nominal 50 Hz on the AgileX backend. The empirical question is whether programmed tools reduce execution-agent interaction costs while maintaining task success. 4 Experiments We evaluate task success, time, and token cost in simulation across execution agents and interfaces, time and token efficiency on a real robot against published step-by-step control, and author-guided tool revision in a case study. Claude Fable 5.1 in Claude Code serves as the programming agent, and Claude Opus 5.5 in Claude Code for the task-specific pouring tool of pour balls; the execution agent is named per row, and in simulation four different frozen models take that role. Agent conditions use the API tool-call path; a human operator driving the same tools through the GUI succeeds in 23 of 25 simulated episodes. 5 Table 1. Success and execution cost on five RoboDojo tasks. Upper block: success rates as posted on the public leaderboard [36] (official protocol, not paired with ours, no cost data) for the best learned policy of each of the three teams with the highest five-task mean,π 0.5 , and GPT-6 Astra through the RoboProbe harness [47]. Lower block: five episodes per task (seeds 0–4); success is the official terminal state. Avg. SR: average success rate, the mean of the five per-task rates (tasks weighted equally). Time and Tokens average all episodes, failures included, and count execution-agent output with reasoning. Program: one program over the same tools, written in advance and run without the model (time includes both phases). Pour balls uses its task-specific pouring tool. Bold: best success per task over all rows (ties included); within the leaderboard block and within each agent (separated by rules), the best Avg. SR and the lowest time and tokens. Method Pour liquid into cup Pour balls into vase Fold clothes Swap blocks a Stack blocks Avg. SR↑ Time (min)↓ Tokens (k)↓ π 0.5 28.013.338.70.013.318.7– GPT-6 Astra, RoboProbe8.04.072.00.088.034.4– Xiaomi-Robotics-164.046.073.30.021.340.9– Liber-0 Preview57.031.079.01.072.048.0– Simate-beta57.345.380.038.769.358.1– DeepSeek-V4-Flash, native0.00.00.00.040.08.021.444.3 DeepSeek-V4-Flash + URAI40.060.00.0100.040.048.020.240.3 GPT-6 Astra, native0.00.020.00.060.016.013.86.9 GPT-6 Astra, program40.040.00.00.040.024.07.04.6 GPT-6 Astra + URAI40.060.00.0100.060.052.09.44.7 Claude Fable 5.1, native0.00.020.00.060.016.025.052.8 Claude Fable 5.1 + URAI20.040.020.0100.080.052.017.430.4 Claude Opus 5.5, native40.00.040.00.080.032.017.432.3 Claude Opus 5.5, program0.040.00.020.060.024.015.138.0 Claude Opus 5.5 + URAI40.060.040.080.080.060.013.420.0 a The tools used for swap blocks bundle a task-level place-on-pad procedure with button pressing, so URAI and native control differ in tool semantics as well as call granularity; the program rows share these tools and give the cleaner contrast. 4.1 RoboDojo: five-task success rate Table 1 compares four frozen execution agents, Claude Fable 5.1 and Claude Opus 5.5 in Claude Code, GPT-6 Astra in the Codex CLI, and DeepSeek-V4-Flash in an open-source coding agent, on multiple RoboDojo tasks [6]: Pour Liquid Into Cup, Pour Balls Into Vase, Fold Clothes, Swap Blocks, and Stack Blocks. Each agent controls the same simulator either through URAI tool calls or natively, by commanding fingertip targets with a yaw or, for one arm, any end-effector orientation (so pouring is possible natively), and gripper values. Each episode starts a new agent session in a fresh working directory, isolated from memory, ledgers, and other episodes; the agent sees only the camera images and a public depth probe, chooses pixels and arguments itself, and only RoboDojo’s own terminal state counts. Each task has one episode per seed 0–4 with the same thirteen reusable tools, frozen before evaluation; for pour balls, a first round with the reusable pouring tool prompted a task-specific pouring tool, developed on separate seeds, with which all its URAI and program episodes were rerun. The agents mainly used the pouring tool, the PnP tool (stack, and its revised version of Section 4.3 for fold), and place-on-pad with button pressing (swap). Protocol. Both conditions use the same task-specific simulator-step budgets (400, 600, 500, 700, and 550, in table order) and a 100-command limit. Agents stop when they judge the task complete or impossible, lack steps for another motion, or reach the command limit, then return both arms home once for official scoring. Wall-clock limits are 45 minutes for URAI and 60 for native control; agent failures and tasks unfinished at the limit count as failures; one URAI episode that had already succeeded and returned home when its limit expired counts as a success. Time spans session launch to exit; tokens count execution-agent output, including reasoning where reported. Results. Through URAI, every agent succeeds more often than under native control (52 vs 16% for Claude Fable 5.1, 60 vs 32% for Claude Opus 5.5, 52 vs 16% for GPT-6 Astra, and 48 vs 8% for DeepSeek-V4-Flash; 53.0% against 18.0% overall), and three of them finish an episode 1.3–1.5×faster with 1.5–1.7×fewer output tokens, whereas DeepSeek-V4-Flash saves only about 5% of its time and 9% of its tokens. Paired by agent, task, and seed, only URAI succeeds in 40 of 100 pairs and only native control in 5 (exact two-sided McNemarp = 7.9× 10 −8 ). The largest single gap is swap blocks, solved in 19 of 20 URAI episodes and in none of the 20 native ones; without it, URAI still leads for every agent, 42.5% against 22.5% (21 against 5 discordant pairs,p = 0.0025). A program written in advance over the same tools isolates the agent’s decisions between calls: in this control, Claude Opus 5.5 or GPT-6 Astra first inspects the scene with observations, depth probes, and dry runs, then writes one program that calls the tools and branches on their 6 returns, and the program runs without the model. It succeeds in 12 of 50 episodes, against 28 of 50 when the agent decides after each call (17 against 1 discordant pairs,p = 1.4× 10 −4 ), and in 11 against 19 of 40 without swap blocks (8 against 0,p = 0.0078). Deciding between calls, not the tool collection alone, therefore accounts for part of the gain. For reference, the best learned policy on the leaderboard averages 58.1% (63.0% against Claude Opus 5.5’s 55.0% without swap blocks), and GPT-6 Astra averages 34.4% through the RoboProbe harness against 52.0% through URAI. Where execution time is spent. Table 2 shows that URAI replaces many short motion calls with fewer, longer ones: 3.5–10.0 calls per episode lasting 29–72 s each, versus 21.0–25.6 calls of 19–20 s natively. For Fable, Opus, and Astra, time outside robot and observation/probe commands falls by about 45%, 38%, and 32%, respectively, consistent with less time spent deciding between motions. DeepSeek shows the opposite pattern: although its motion-call count falls from 25.6 to 10.0, time outside commands rises from 441 to 509 s, and observations and probes take a larger share of the episode. Coarser motion calls therefore do not ensure lower total cost; the execution agent’s observation and decision behavior also matters. 4.2 Real-robot evaluation The real-robot platform is our AgileX PiPER-X dual-arm system with four RealSense RGB-D cameras. Seven tabletop tasks are run (Figure 3), all with GPT-6 Astra as the execution agent calling URAI tools. Unscrew bottle cap and tic-tac-toe correspond to GPT-Policy’s dual-arm tasks R3 (its best condition) and R8 (human interaction, same success criterion) [8], and block into bowl repeats Robocurve’s task, run there under direct end-effector control [35]. Hot-dog serving, tossing blocks into a bowl, pouring blocks, and clothes folding have no published same-task reference and are reported for URAI alone (lower block). Measurement protocol. The URAI tasks with published references run at1×speed; time spans task release to confirmed completion or failure, including robot motion, retries, and waiting for human moves in tic-tac-toe. Output tokens include reasoning; time, tokens, and calls average all three trials, failures included. Robocurve uses YAM arms, a 25% speed cap, and a 20-call budget over 20 trials per task. Results.All three tasks with a published reference succeed in 3/3 trials (Table 3). In tic-tac-toe, the robot lost none of three games against a human (5.0 minutes, 37.0 model calls, and 3.26k output tokens per game), against GPT-Policy’s 13.6 minutes and 69.7 decisions with the same model: 2.7×faster at the same 3/3 success. Unscrew bottle cap took 5.1 minutes and 24.7 calls per trial against R3’s 17.9 minutes and 54.7 decisions (3.5×faster, also 3/3). Block into bowl took 61.3 s, 6.3 calls, and 443 output tokens per trial against Robocurve’s 2.5 minutes, at most 20 calls, and 2.1k tokens (19/20) with the same model under direct end-effector control: 2.4×faster with 4.7×fewer tokens. The first trial (71.2 s, 731 tokens) also wrote the execution script that the other two reused (51.5 s, 182 tokens; 61.2 s, 417 tokens); on its own it is still 2.1×faster with 2.9×fewer tokens. Among the URAI-only tasks, run ten times each, hot-dog serving and pour blocks succeed in all ten trials, fold clothes in 8/10 (90.0% mean progress), and toss blocks into bowl in 7/10 (91.7%); their token and time columns are means over the first three trials, the only ones with a token log. 4.3 Tool evolution: revising one tool for fold clothes The programming agent, steered by the author in natural language, revised the existing two-point PnP tool of Table 1 seven times (eight versions with the initial one) instead of writing a folding routine. The initial version rejected every click on the shirt as a large rigid object; the revisions add sheet pinching with local thickness checks, synchronized dual-arm transport, outward wrist tilt while carrying, vertical release, and a drop height and clearance taken from depth; for example, replacing the median layer height with its local 0.9 quantile lets the fingers release above an already folded layer. Each version keeps the call signature, ships with unit tests, and has no branch on seed, garment, or task name. The frozen final version is tested by the four frozen execution agents in fresh sessions, choosing pixels themselves: 3 of 20 episodes succeed on seeds 0–4 (Table 1) and 2 of 20 on seeds 5–9, whereas a human operator choosing pixels through the GUI succeeds in 4 of 5. With this tool, choosing pinch and drop points is the agents’ main difficulty: in most failed GPT-6 Astra and Claude Opus 5.5 episodes, the first sleeve pinch sits at the sleeve tip, where the fingers close on nothing. 7 Table 2. Execution-time decomposition on RoboDojo. Motion and observation/probe columns give mean per-episode shares of wall time; Other is time outside logged commands. All evaluated episodes, including failures, are included. Bold: within each agent, the fewer motion calls and the less time outside logged commands; the other columns describe the composition of an episode. Condition Motion calls↓ s/call Motion (%) Obs./probe (%) Other (%)↓ Other (s)↓ Claude Fable 5.1 + URAI4.363313138426 Claude Fable 5.1, native22.820341651772 Claude Opus 5.5 + URAI3.572373528250 Claude Opus 5.5, native21.820441838405 GPT-6 Astra + URAI5.644472133188 GPT-6 Astra, native21.019501634276 DeepSeek-V4-Flash + URAI10.029313436509 DeepSeek-V4-Flash, native25.619402634441 Seconds per call pools motion calls; other entries are per-episode means. Table 3. Real-robot results. Upper block: tasks with a published GPT-6 Astra reference; URAI reports three trials per task with mean time, output tokens, and model calls per episode; the references run on other hardware. Lower block: tasks without a reference; progress is the fraction completed (hot-dog: items of four; toss blocks: blocks of six; fold clothes: operator score); success requires the whole task. Bold: the better value within each task where both rows report a comparable measure (success only at equal trial counts; a call budget is not a measured count). TaskMethodSuccess↑ Decisions / model calls↓ Tokens↓ Time (min)↓ Unscrew bottle cap GPT-Policy, published (R3)3/354.7–17.9 GPT-6 Astra + URAI3/324.71.52k5.1 Tic-tac-toe GPT-Policy, published (R8)3/369.7–13.6 GPT-6 Astra + URAI3/337.03.26k5.0 Block into bowl Robocurve, published19/20≤20 (budget)2.1k2.5 GPT-6 Astra + URAI3/36.34431.0 URAI-only tasks, ten trials each (success; mean progress); cost over the first three trials Hot-dog servingGPT-6 Astra + URAI10/10 (100%)–2.40k7.9 Toss blocks into bowlGPT-6 Astra + URAI7/10 (91.7%)–4.69k8.1 Pour blocksGPT-6 Astra + URAI10/10 (100%)–2.29k3.1 Fold clothesGPT-6 Astra + URAI8/10 (90.0%)–1.41k4.1 URAI calls count execution-agent model responses, including observations, polling, and recovery; GPT-Policy counts robot-tool requests;≤20 is a call budget. The first bottle-cap trial excludes unmetered diagnosis and repair. –: not reported. 5 Limitations and Discussion Our measurements cover five simulated and seven real-robot tasks with few episodes per cell; under our protocol, with this tool collection and task mix, all four execution agents have higher aggregate success, and three of them lower execution-phase time and token cost. Pooled over agents, the success gain is significant with and without swap blocks (McNemarp = 7.9× 10 −8 andp = 0.0025), but five episodes per cell do not establish per-task differences. The program control covers two agents on seeds 0–4; no prompt- or memory-only revision control was run. The real-robot comparisons are against published runs on other hardware with no same-robot step-by-step baseline, so the reported ratios are not causal estimates of a speedup. Tool programming and the author’s steering of it cost unmetered time and tokens that may offset the execution-phase savings; the interface is validated on one simulator and one position-controlled dual-arm platform without force regulation, so portability is not established. 6 Conclusion Code as policy with the agent in the loop pairs agent-written tools with feedback-driven tool selection through URAI, which humans share. On five RoboDojo tasks, four frozen execution agents succeed more often through URAI than under direct fingertip control, and deciding after each call succeeds more often than a program written in advance over 8 t1t2t3t4t5t6 Tic-tac-toe PnP(...) Unscrew bottle cap bottle_cap_twist(...) Block into bowl PnP(...) Hot-dog serving PnP(...) Toss blocks toss(...) Pour blocks pour(...) Fold clothes PnP(...) first frame = initial scene, last frame = task completed Figure 3. Key frames of the seven AgileX tasks under URAI with GPT-6 Astra, initial scene (t1) to completion (t6); chips name the tool called. Fold clothes is the demonstration-video run; other rows are evaluated trials. the same tools; three of the four are also faster and use fewer output tokens. Next, we will extend the tool collection to more tasks and robot platforms and meter tool-authoring cost alongside execution cost. References [1]Reyna Abhyankar, Qi Qi, and Yiying Zhang. OSWorld-Human: Benchmarking the Efficiency of Computer-Use Agents, 2025. URL https://arxiv.org/abs/2506.16042. [2]Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, and Xin Eric Wang. Agent S: An Open Agentic Framework that Uses Computers Like a Human, 2024. URL https://arxiv.org/abs/2410.08164. [3]Diogo Almeida. Introducing System One Models & Jev. TypeSafe AI blog, 2026. URLhttps://typesafe. ai/blog/introducing-system-one-models-and-jev. Published September 15, 2026. Accessed September 21, 2026. [4] James E. Bobrow, Steven Dubowsky, and J. S. Gibson. Time-optimal control of robotic manipulators along specified paths. The International Journal of Robotics Research, 4(3):3–17, 1985. [5]Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou. Large language models as tool makers. In International Conference on Learning Representations, 2024. URL https://arxiv.org/abs/2305.17126. [6] Tianxing Chen, Yue Chen, Zixuan Li, Junyuan Tang, Kailun Su, Haoran Lu, Weijie Wan, Baijun Chen, Songling Liu, Haowen Yan, Honghao Su, Zhiyang Dou, Kaixuan Wang, Dandan Zhang, Yunze Liu, Yan Qin, Qiwei Liang, Qiwei Wu, Zijian Lin, Wenwei Lin, Yuran Wang, Minghua He, Tianshu Wu, Ruihai Wu, Jingquan Zhou, Kai-Chong Lei, Haibao Yu, Yuanfeng Ji, Weiyang Jin, Guanyu Lin, Xiaofan Li, Qi Xiong, Renjing Xu, Zhongyu Li, Wenhao Chai, Enze Xie, Ziwei Wang, Yao Mu, Hao Dong, Wojciech Matusik, Mingyu Ding, Wenbo Ding, Ping Luo, and Masayoshi Tomizuka. RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies, 2026. URL https://arxiv.org/abs/2607.04434. [7] Yanzhe Chen, Zechen Bai, Zhijun Cao, Wenzheng Zeng, Kevin Qinghong Lin, Yiqi Lin, Guoqiang Liang, Kevin Yuchen Ma, Qiming Huang, and Mike Zheng Shou. Show-Harness: Just a VLM Agent Can Play Robots, 2026. URL https://arxiv.org/abs/2609.10522. 9 [8] Dongzhou Cheng, Taoran Yi, Ye Fang, Xingwu Zhang, Fan Feng, Yixuan Li, Gengxiong Zhuang, Rongze Wang, Shuai Yang, Wei Song, Weizhi Xue, Minyan Wu, Jie Gui, Jiaqi Wang, and Tong Wu. In-context robot learning with vlm agents, 2026. URL https://arxiv.org/abs/2609.19138. [9]Cheng Chi, Zhenjia Xu, Chuer Pan, Eric Cousineau, Benjamin Burchfiel, Siyuan Feng, Russ Tedrake, and Shuran Song. Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots, 2024. URL https://arxiv.org/abs/2402.10329. [10]Karim Elmaaroufi, Justin Svegliato, Sarunas Kalade, Graham Schelle, Sanjit A. Seshia, and Matei Zaharia. RHO: Your Coding Agent is Secretly a Roboticist, 2026. URL https://arxiv.org/abs/2606.16458. [11]Letian Fu, Justin Yu, Karim El-Refai, Ethan Kou, Haoru Xue, Huang Huang, Wenli Xiao, Guanzhi Wang, Dantong Niu, Fei-Fei Li, Guanya Shi, Jiajun Wu, Shankar Sastry, Yuke Zhu, Ken Goldberg, and Linxi "Jim" Fan. CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation, 2026. URL https://arxiv.org/abs/2603.22435. [12]Shijia Ge, Yijun Liu, Yinxin Zhang, Shuzhao Xie, Weixiang Zhang, Mingcai Zhou, and Zhi Wang. VGGT-DP: Generalizable Robot Control via Vision Foundation Models, 2026. URL https://arxiv.org/abs/2509.18778. [13] Jiayuan Gu, Sean Kirmani, Paul Wohlhart, Yao Lu, Montserrat Gonzalez Arenas, Kanishka Rao, Wenhao Yu, Chuyuan Fu, Keerthana Gopalakrishnan, Zhuo Xu, Priya Sundaresan, Peng Xu, Hao Su, Karol Hausman, Chelsea Finn, Quan Vuong, and Ted Xiao. RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketches, 2023. URL https://arxiv.org/abs/2311.01977. [14]Meng-Hao Guo, Zhe-Han Mo, Jia-Jun Wang, Yi Zhang, Kejin Wang, Yi-Xuan Deng, Jia-Peng Zhang, Yongming Rao, and Shi-Min Hu. Transferring the intelligence of vlms to robotic control. Technical Report, 2026. URL https://robodawn.top/RoboDawn.pdf. Technical report PDF from the project page. Accessed September 21, 2026. [15]Hengyuan Hu, Priya Sundaresan, Jensen Gao, and Dorsa Sadigh. Via: Visual interface agent for robot control, 2026. URL https://arxiv.org/abs/2607.11119. [16] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner monologue: Embodied reasoning through planning with language models. In Conference on Robot Learning, 2022. URL https://arxiv.org/abs/2207.05608. [17]Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models. In Proceedings of The 7th Conference on Robot Learning, volume 229 of Proceedings of Machine Learning Research, pages 540–562. PMLR, 2023. URL https://proceedings.mlr.press/v229/huang23b.html. [18]Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang, and Li Fei-Fei. ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation. In Proceedings of The 8th Conference on Robot Learning, volume 270 of Proceedings of Machine Learning Research, pages 4573–4602. PMLR, 2025. URL https://proceedings.mlr.press/v270/huang25g.html. [19]Yuzhi Huang, Jie Wu, Weijue Bu, Ziyi Xiong, Gaoyang Jiang, Ye Li, Kangye Ji, Shuzhao Xie, Yue Huang, Chenglei Wu, Jingyan Jiang, and Zhi Wang. RoboStream: Weaving Spatio-Temporal Reasoning with Memory in Vision-Language Models for Robotics, 2026. URL https://arxiv.org/abs/2603.12939. [20] Yuka Iwanaga, Masayoshi Tsuchinaga, Kosei Tanada, Yuji Nakamura, Takemitsu Mori, and Takashi Yamamoto. Sketch Interface for Teleoperation of Mobile Manipulator to Enable Intuitive and Intended Operation: A Proof of Concept, 2025. URL https://arxiv.org/abs/2505.13931. [21]Mengzhao Jia, Yang Lin, Xixin Zhang, Zhihan Zhang, Xiaobai Liu, and Meng Jiang. Agent as Policy for Robotic Manipulation, 2026. URL https://arxiv.org/abs/2609.12541. [22]Kamal Kant and Steven W. Zucker. Toward efficient trajectory planning: The path-velocity decomposition. The International Journal of Robotics Research, 5(3):72–89, 1986. 10 [23]Andrej Karpathy. There’s a new kind of coding I call “vibe coding”. Post on X, February 2025. URLhttps: //x.com/karpathy/status/1886192184808149383. Posted February 2, 2025. Accessed September 22, 2026. [24]Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan P Foster, Pannag R Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. Openvla: An open-source vision-language-action model. In Proceedings of The 8th Conference on Robot Learning, volume 270 of Proceedings of Machine Learning Research, pages 2679–2713, 2025. URL https://proceedings.mlr.press/v270/kim25c.html. [25]Yi Li, Yuquan Deng, Jesse Zhang, Joel Jang, Marius Memmel, Raymond Yu, Caelan Reed Garrett, Fabio Ramos, Dieter Fox, Anqi Li, Abhishek Gupta, and Ankit Goyal. HAMSTER: Hierarchical Action Models For Open-World Robot Manipulation, 2025. URL https://arxiv.org/abs/2502.05485. [26]Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as Policies: Language Model Programs for Embodied Control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9493–9500. IEEE, 2023. doi: 10.1109/ICRA48891.2023.10160591. URL https://doi.org/10.1109/ICRA48891.2023.10160591. [27]Fangchen Liu, Kuan Fang, Pieter Abbeel, and Sergey Levine. MOKA: Open-World Robotic Manipulation through Mark-Based Visual Prompting, 2024. URL https://arxiv.org/abs/2403.03174. [28]Yijun Liu, Yuwei Liu, Yuan Meng, Jieheng Zhang, Yuwei Zhou, Ye Li, Jiacheng Jiang, Kangye Ji, Shijia Ge, Zhi Wang, and Wenwu Zhu. Spatial Policy: Guiding Visuomotor Robotic Manipulation with Spatial-Aware Modeling and Reasoning, 2025. URL https://arxiv.org/abs/2508.15874. [29] Yuke Liu. RoboJEV: Two-Stage JEV Control of a Franka Panda in MuJoCo. GitHub repository, Apache-2.0, 2026. URL https://github.com/lykycy123/RoboJEV. Accessed September 21, 2026. [30]Runyu Lu, Yubo Wu, Ethan Kou, Letian Fu, Wenli Xiao, Ajay Mandlekar, Yinzhen Xu, Guanya Shi, Ken Goldberg, Ang Chen, Mosharaf Chowdhury, Yuke Zhu, Linxi "Jim" Fan, and Guanzhi Wang. ASPIRE: Agentic /Skills Discovery for Robotics, 2026. URL https://arxiv.org/abs/2607.00272. [31] Soroush Nasiriany, Fei Xia, Wenhao Yu, Ted Xiao, Jacky Liang, Ishita Dasgupta, Annie Xie, Danny Driess, Ayzaan Wahid, Zhuo Xu, Quan Vuong, Tingnan Zhang, Tsang-Wei Edward Lee, Kuang-Huei Lee, Peng Xu, Sean Kirmani, Yuke Zhu, Andy Zeng, Karol Hausman, Nicolas Heess, Chelsea Finn, Sergey Levine, and Brian Ichter. PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs, 2024. URLhttps://arxiv.org/abs/2402. 07872. [32]OpenAI. Gpt-6 astra: A new generation of intelligence. Official model announcement, 2026. URLhttps: //openai.com/index/gpt-6-astra/. [33]Hung Pham and Quang-Cuong Pham. A new approach to time-optimal path parameterization based on reachability analysis. IEEE Transactions on Robotics, 34(3):645–659, 2018. [34]Thomas Ricouard. Architectural visualization with astra. OpenAI Developers case study, 2026. URLhttps: //developers.openai.com/blog/architectural-visualization-with-astra. [35]Robocurve. GPT-6 Astra on robotic manipulation. Independent experimental report, September 2026. URL https://openai.robocurve.org/gpt-6-astra/. Published September 4, 2026. Accessed September 15, 2026. [36]RoboDojo Team. RoboDojo-Sim public leaderboard: per-task success rates. Leaderboard snapshot dated 2026-09- 23, 2026. URL https://robodojo-benchmark.com/leaderboard. Accessed September 23, 2026. [37]Jiayi Su, Yixin Zheng, Mi Yan, Li Yi, Zhizheng Zhang, and He Wang. GPT 6 Astra as an embodied policy. Technical report and code, 2026. URLhttps://github.com/anonymous-report-421/eval-of-gpt-6-astra-as-policy. Companion report. Accessed September 15, 2026. [38] Priya Sundaresan, Hengyuan Hu, Quan Vuong, Jeannette Bohg, and Dorsa Sadigh. What’s the Move? Hybrid Imitation Learning via Salient Points, 2024. URL https://arxiv.org/abs/2412.05426. [39] Huajie Tan, Peterson Co, Yijie Xu, Shanyu Rong, Yuheng Ji, Cheng Chi, Xiansheng Chen, Qiongyu Zhang, Zhongxia Zhao, Pengwei Wang, Zhongyuan Wang, and Shanghang Zhang. Action-Sketcher: From Reasoning to 11 Action via Visual Sketches for Long-Horizon Robotic Manipulation, 2026. URLhttps://arxiv.org/abs/2601. 01618. [40]Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. Transactions on Machine Learning Research, 2024. URL https://arxiv.org/abs/2305.16291. [41]Maggie Wang, Lars Osterberg, Stephen Tian, Ola Shorinwa, Jiajun Wu, and Mac Schwager. InSight: Self-Guided Skill Acquisition via Steerable VLAs, 2026. URL https://arxiv.org/abs/2606.24884. [42]Wenli Xiao, Jia Xie, Tonghe Zhang, Haotian Lin, Letian "Max" Fu, Haoru Xue, Jalen Lu, Yi Yang, Cunxi Dai, Zi Wang, Jimmy Wu, Guanzhi Wang, S. Shankar Sastry, Ken Goldberg, Linxi "Jim" Fan, Yuke Zhu, and Guanya Shi. ENPIRE: Agentic Robot Policy Self-Improvement in the Real World, 2026. URLhttps: //arxiv.org/abs/2606.19980. [43]Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments, 2024. URL https://arxiv.org/abs/2404.07972. [44]Haoqi Yuan, Zhixuan Liang, Anzhe Chen, Ye Wang, Haoyang Li, Pei Lin, Yiyang Huang, Zixing Lei, Tong Zhang, Jiazhao Zhang, Jie Zhang, Jingyang Fan, Gengze Zhou, Qihang Peng, Chenxu Lv, Xiaoyue Chen, An Yang, Fei Huang, Junyang Lin, Dayiheng Liu, Jingren Zhou, Chenfei Wu, and Xiong-Hui Chen. Qwen- RobotManip Technical Report: Alignment Unlocks Scale for Robotic Manipulation Foundation Models, 2026. URL https://arxiv.org/abs/2606.17846. [45]Lifan Yuan, Yangyi Chen, Xingyao Wang, Yi R. Fung, Hao Peng, and Heng Ji. CRAFT: Customizing LLMs by creating and retrieving from specialized toolsets. In International Conference on Learning Representations, 2024. URL https://arxiv.org/abs/2309.17428. [46]Tianhao Zhang, Zoe McCarthy, Owen Jow, Dennis Lee, Xi Chen, Ken Goldberg, and Pieter Abbeel. Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation, 2018. URLhttps: //arxiv.org/abs/1710.04615. [47]Wenbo Zhang, Kaixuan Wang, Yutao Ouyang, Xiaoyu Huang, Liyang Li, Kailun Su, Weiyang Jin, Wenhao Chai, Haotian Liang, Zhiyang Dou, Yue Chen, and Tianxing Chen. An Unexpected Robot Policy: Early Evaluations of GPT-6 Astra on RoboDojo and Beyond. RoboDojo project report, 2026. URLhttps://robodojo-benchmark. com/report/gpt-6-astra-eval. Site snapshot dated 2026-09-21. Accessed September 22, 2026. [48]Tony Z. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware, 2023. URL https://arxiv.org/abs/2304.13705. [49]Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. WebArena: A Realistic Web Environment for Building Autonomous Agents, 2024. URL https://arxiv.org/abs/2307.13854. 12 A Extended Related Work This appendix gives the full discussion that Section 2 condenses, preceded by the background on frontier models that the introduction summarizes. Frontier models in spatial software work and physical manipulation.Frontier models are becoming capable of complex spatial work through computer and software tools. OpenAI reports that GPT-6 Astra reconstructs three-dimensional objects from multiview renders by generating CAD code, reaching 95.9% geometric overlap on BenchCAD [32]. In a documented design workflow, Astra created an editable house scene in Blender, inspected the application through computer use and rendered previews, and transferred the scene into a walkable Unreal Engine 5 environment; the same account also describes reference-guided garden and other 3D scene construction [34]. These examples demonstrate visual inspection, spatial reasoning, and iterative tool use, although the Blender geometry was produced primarily through scripts rather than mouse-only GUI control. They do not by themselves establish physical manipulation performance. Frontier models such as GPT-6 Astra have also demonstrated the ability to complete physical manipulation tasks by selecting end-effector targets and gripper commands from visual observations [35]. In this form of control, the model inspects the scene, chooses a Cartesian pose and gripper state, executes the command through a robot interface, and observes the outcome before deciding what to do next. The robot’s inverse kinematics and local controller realize each command. These results establish that general-purpose models can make useful manipulation decisions on real robots without task-specific training of the frontier model. In this setting the model itself is the policy, one motion at a time; GPT-Policy studies it on real robots with in-context demonstrations and human interaction [8]. Frontier models as robot decision makers.Code as Policies establishes program generation as an interface to perception and control: the model writes the program first, generating undefined functions hierarchically, and the program can call perception APIs and implement feedback loops and waypoint policies [26]; at run time its code, not the model, decides what runs next and handles only the cases it checks. VoxPoser uses language models to compose three-dimensional value maps [17]; MOKA uses visual marks to specify affordances and motion [27]; and PIVOT uses iterative visual prompting for action selection [31]. ReKep generates constraints that a local optimizer converts into reactive robot motion [18]. These approaches already provide alternatives to predicting each motor command with a frontier model. CaP-X further examines the effects of API abstraction and interaction, adding multi-turn interaction with structured execution feedback and visual differencing, and automatic skill synthesis in CaP-Agent0, and exposing a trade-off between the convenience of higher-level primitives and the expressivity of lower-level control [11]. Agent as Policy drives a physical robot with a general-purpose coding agent that writes programs, issues motion commands, and revises them from physical feedback; a preparation agent authors reusable task definitions once per task type and an execution agent runs each trial in a fresh session, with model weights fixed [21]. RoboStream is a training-free pipeline that turns each observation into an object-centric scene graph, keeps a short-term causal memory across steps, prompts a frozen VLM for the next manipulation step, and executes the grounded step as a target pose in closed loop [19]. These works place the policy in a program whose code makes the run-time decisions, in a composed objective, or in the model’s own step-by-step choices; GPT-Policy takes the last route with a frontier VLM on real robots [8]. We place it in a persistent collection of executable tools that coding agents write, validate, and revise across episodes, in the manner of vibe coding [23], while the frontier model at run time chooses and parameterizes complete tools and decides again after every tool execution, rather than leaving later decisions to generated code. URAI is the interface that makes this possible: an editable motion contract whose visual and programmatic forms share execution semantics, so that humans and agents operate the same tools. Our experiments compare it with direct fingertip-target control in the same simulator, with a Code-as-Policies-style condition in which the agent writes one program over the same tools that then makes the subsequent calls without the model (Appendix C.1), and with published step-by-step runs on real robots. Persistent program and policy improvement.ASPIRE autonomously writes and repairs code-as-policy robot programs from multimodal execution traces, retains validated repairs in a reusable skill library, and uses evolutionary search to expand the explored task and program space [30]. ENPIRE instead provides a real-world experimentation harness in which coding agents operate a repeatable reset–execute–verify–refine loop and improve heuristic or learned policies from physical rollouts [42]. RHO has tool-enabled coding agents search, before deployment and with reflective feedback from reward and execution in simulation, for multi-file policy repositories that compose perception, planning, and control primitives; on its manipulation benchmarks the selected repository then runs single-turn, without LLM code edits at deployment, and on RAI’s O3DE benchmark, where an LLM stays in the control loop, RHO optimizes that agent’s 13 harness of prompts, tools, and control code [10]. All of its benchmarks are simulated, and it meters the token cost of the search. These systems are close precedents for persistent improvement outside foundation-model weights. Outside robotics, Voyager accumulates a library of code skills for an embodied language agent [40], and LATM and CRAFT separate a tool-making model from a tool-using model [5,45]. URAI focuses on a complementary interface question. Its tools are called at run time by a frozen execution agent that reads feedback between calls; humans operate the same tools through a GUI with the same execution semantics; tool revisions are proposed by a programming agent steered in natural language, validated on development episodes, and kept across episodes, rather than found by reward-driven search; and it is evaluated on a real robot as well as in simulation. Recent frontier-model control evaluations.Robocurve evaluates GPT-6 Astra on physical YAM arms through absolute end-effector pose commands, using three camera views, proprioception, a 20-call budget, and a 25% speed cap [35]. These settings matter when interpreting its time and token measurements. GPT-Policy uses an off-the-shelf VLM with a constrained robot-tool interface and tests human videos, robot demonstrations, goal images, interaction history, and online human input on real robots [8]. We run two of its stationary tabletop tasks on our AgileX dual-arm platform and exclude its mobile exploration task. Su et al.[37]compare direct control with hybrid control that accepts or corrects π 0.5 proposals. Their RoboDojo implementation executes 1–5 steps for direct commands or corrections, versus 1–15 steps when accepting policy actions. Action priors and segment lengths therefore change together; the token savings do not isolate an interface effect. Their reported physical duration excludes model-response latency. RoboDawn gives a frozen VLM a compact interface of discrete translation, rotation, and gripper commands, supplemented by optional in-context demonstrations [14]. It reports RoboDojo [6] and RoboTwin 2.0 evaluations and physical Franka and Piper trials. Its published RoboDojo aggregate covers 42 tasks and cannot be treated as a five-task URAI result. Its released one-demonstration run on our five layouts is a published reference under another protocol, not a control of our study; likewise, GPT-Policy’s and Robocurve’s physical runs are published references from other hardware, which we compare with URAI runs on our AgileX platform (Section 4.2). Action tokens and execution granularity.OpenVLA casts visuomotor control as language-model prediction of tokenized robot actions: continuous action dimensions are discretized into a finite vocabulary and decoded back to executable values [24]. Qwen-RobotManip scales the learned-policy route without action tokens: a Qwen3.5-4B backbone with a flow-matching action expert, pretrained on roughly 38,100 hours of robot, human-video, and synthesized data, emits continuous action chunks at 30 Hz [44]. Learned visuomotor policies also draw on spatial priors: VGGT-DP adopts the VGGT 3D foundation model as the visual encoder of a diffusion policy with proprioception-guided visual learning [12], and Spatial Policy conditions embodied video generation on an explicit spatial plan, predicts actions with a flow-based module, and replans from spatial reasoning feedback [28]. RoboDawn’s textualmove,rotate, andgrippercommands revive the broader idea of discrete action symbols for a frontier VLM, but they are manually specified semantic primitives rather than OpenVLA’s learned, quantized per-dimension motor tokens [14]. Show-Harness takes the same route with a fixed vocabulary of directional moves, axis rotations, and gripper commands; embodiment-specific interpreters map each unit deterministically to one Cartesian pose increment, and the vocabulary is driven zero-shot by Gemini-3.1 Pro or by a LoRA-tuned Qwen3.5-2B on Franka and bimanual AgileX arms [7]. URAI also presents discrete tool calls to the execution agent, but the programming agent can provide both reusable tools such as PnP and task-specific executable procedures. Their internal motion phases execute locally. Thus the distinction is not discrete versus continuous output alone; it is the duration and responsibility assigned to one model-level decision, together with the cost of authoring the tool. Our native condition, in which the same agents command fingertip targets in the same simulator, and our program condition, in which a program written in advance over the same tools makes the calls, place this boundary differently (Appendix C.1). Sketches and intermediate representations for manipulation. Trajectory sketches are established robot instructions. RT-Trajectory conditions a learned policy on rough trajectories provided by humans, videos, or model-generated specifications [13]. Its representation already includes gripper transitions and temporal information; shared human/model input and time-annotated trajectories are therefore precedents, rather than standalone novelty claims for URAI. HAMSTER trains a high-level VLM to predict image-space paths and a low-level policy to execute them from three-dimensional observations [25]. Action-Sketcher incorporates editable visual sketches into a learned reasoning-and- action framework [39]. SPHINX combines sparse waypoint actions and dense local control, with a web interface for collecting salient points and waypoints [38]. VIA instead lets off-the-shelf frontier agents control a robot through a browser-based three-dimensional interface [15]; it is an important visual-interface precedent for the native conditions, 14 although its published LIBERO and assembly results are not RoboDojo scores. Our focus is the interface used by off-the-shelf agents and the cost of completing physical tasks through it. We make metric geometry and timing explicit at the command boundary. Human–robot sketch interfaces and demonstration collection.Iwanaga et al.[20]study a web-based sketch interface for a mobile manipulator and report lower workload and greater intuitiveness than a conventional axis-control interface in their user study. This work directly motivates evaluating the usability of drawing-based commands. Other interfaces capture detailed human behavior through different hardware: ALOHA supports bimanual teleoperation for imitation learning [48], VR teleoperation provides pose-based demonstrations [46], and UMI uses handheld grippers and a carefully designed policy interface to transfer demonstrations to robots [9]. These systems are relevant precedents for expressivity and data acquisition, but their reported results do not establish a common ranking of interface precision or user friendliness. Show-Harness’s GUI Manipulation Interface is closer to our setting: humans and VLMs issue the same single-step semantic units through labeled controls and keystrokes, and each recorded observation–action pair is reusable across embodiments [7]. URAI likewise offers one software action vocabulary to human and model operators, but each entry is a complete, timed tool with explicit metric arguments rather than a fixed increment. Computer-use interfaces and efficiency.WebArena exposes browser observations as screenshots, DOM representations, or accessibility trees, and supports actions grounded by coordinates or element identifiers [49]. OSWorld evaluates agents in real desktop applications using execution-based task outcomes [43]. Agent S explicitly studies an agent–computer interface within a computer-use framework [2]. These works motivate treating the observation/action boundary as part of an agent system. The robotics setting adds a crucial distinction: physical geometry must be estimated from sensors, and a command’s outcome depends on continuous dynamics and contact. URAI adopts the interface-design perspective without assuming that physical scenes provide native, error-free accessibility trees. OSWorld-Human motivates measuring inference and interaction efficiency in addition to success [1]. In robotics, InSight reports reduced VLM thinking time through acquired primitives [41]. Jev shortens the decision itself: a non-autoregressive model answers bounded typed questions in one pass at 70–500 ms end-to-end [3], and RoboJEV drives a MuJoCo Franka with two such choices per 1 cm step, reporting 43/50 physics-checked successes over five tasks with ten seeds each, against 48/50 for a same-scene rule baseline, and no latency or token figures [29]. These systems lower the cost of each decision. URAI instead targets the number of decisions per episode: a programming agent may revise reusable or task-specific tools while a separate execution agent invokes a frozen tool version during evaluation. Table 4. Closest related systems on the dimensions along which URAI differs, as described in the respective reports. SystemWho writes the code, and when Unit of one model decision Model at run timePersistence across episodes Human interface Code as Policies [26]LM, one program per instruction the whole programprogram decides; perception calls inside none– CaP-X / CaP-Agent0 [11] coding agent, one program per turn one program per turnre-plans between turns from execution feedback synthesized skills (CaP-Agent0) – Agent as Policy [21]preparation agent per task type; execution agent per trial program and motion commands revises from physical feedback saved procedures reused– ASPIRE [30]agent writes and repairs programs between rollouts programprogram runsvalidated skill library– ENPIRE [42]coding agent revises the policy between rollouts policypolicy runsretained policiesharness LATM / CRAFT [5,45]tool-maker LLM, offlineone tool call by the tool user tool user decides each calltoolset– RoboDawn [14], Show-Harness [7] fixed command vocabulary, hand-specified one increment (move, rotate, gripper) VLM decides every stepin-context demonstrations Show-Harness GUI shares the units URAIprogramming agent, author-steered, between episodes; frozen before evaluation one complete, timed toolexecution agent decides after every tool tool collection, code, and strategy GUI with the same tool contract B Implementation Details This appendix completes the interface description of Section 3. Figure 2 in Section 3 details the two loops that Figure 1 summarizes. 15 Tool calls.The execution agent lists the tool collection, reads a tool’s description and argument schema (names, ranges, and defaults), and calls one tool at a time with an observation identifier, image pixels chosen from that observation, and its own arguments. An observation identifier stays valid for 120 s and only until the robot next moves; a call that cites an expired identifier is refused as stale. A tool plans all of its stages before it moves. A dry run returns that plan without motion; a refusal before motion reports which stage failed and why, and costs no simulator steps. After a tool finishes or is interrupted, its reply reports the stages it completed, which is not a declaration that the task is complete; because an identifier expires with the next motion, every call after a motion is grounded in a new observation. Path inputs and timing rules. The prototype accepts sampled control points and bounded mathematical expressions. Rotations are interpolated onSO(3)rather than by independently interpolating Euler angles. The nominal relation in Equation(2)covers regular translating segments. Pure rotations, endpoint ramps, and stationary holds have separate timing rules. The implementation also accounts for angular speed and robot-dependent motion limits, so a requested speed specifies the generated reference. Gripper events. Gripper events are associated with progress and execution conditions. For ordinary placement, the sequence is descent to the placement pose, satisfaction of the configured arrival condition, gripper opening, and retreat. Dynamic release uses a different event semantics, with dispatch tied to motion progress and a calibrated release delay. Visual grounding, GUI, and API.A valid depth observation at image pixel ̃u = (u,v, 1) ⊤ is back-projected using the camera intrinsics K and camera-to-world transform (R wc ,t wc ): p w = R wc D(u,v)K −1 ̃u + t wc .(3) This expression locates an observed surface point. It does not determine the desired height of an airborne motion at the same pixel. URAI therefore distinguishes surface-grounded input from free-space path editing. Depth-derived orthographic side views expose height as an editable coordinate, while a three-dimensional preview relates the edited path to the observed scene. The GUI supports continuous drawing, arm selection, orientation handles, and a speed-profile editor. Explicit gripper events allow path geometry and grasp/release decisions to be edited together. Higher-level drawing modes generate approach, grasp, transport, and placement phases from sparse input, and a human operator can call the tools of the collection with clicked pixels. The programmatic interface exposes corresponding control points, profiles, motion settings, and tool calls. Expressions are evaluated in a bounded arithmetic interpreter. This route allows an agent to specify an entire curve without issuing a mouse action for each point. Equivalent visual and API requests use the same backend and the same execution semantics; Appendix C.1 lists the images, depth probe, and robot state that each RoboDojo condition receives. Local execution backend. If the requested start differs from the current end-effector pose, the backend constructs an approach before the user-specified motion. Planning checks the full approach and task trajectory against the configured kinematic and geometric constraints. The dual-arm implementation can bring selected arms to their start poses before beginning their task trajectories together. Observations and previews are versioned so that execution uses a plan associated with the current robot state. Changed state requires refreshing and rebuilding the plan. On the AgileX dual-arm backend, the implementation streams references locally at a nominal 50 Hz reference-dispatch frequency in joint-position mode. Its model-based checks cover the robot and the calibrated table, and gripper feedback provides an execution signal. C Experimental Setup This appendix gives the configuration of the simulated evaluation of Section 4.1 and of the real-robot evaluation of Section 4.2. 16 C.1 RoboDojo Tasks. The five tasks, with RoboDojo’s instruction, its public success criterion, and the episode budget in simulator steps, are Pour Liquid Into Cup (“Pour the liquid from the bottle into the cup”; the cup contains the required amount of liquid and the bottle is upright; 400 steps), Pour Balls Into Vase (“Pour all the balls from the cup into the vase”; all seven balls inside, cup upright, robot back at origin; 600), Fold Clothes (“Fold the clothes neatly”; both sleeves folded inward, hems near the shoulder points, hem line aligned with the shoulder line within the angle threshold, robot back at origin; 500), Swap Blocks (“Swap the two blocks using the empty mat, pressing the button after each move”; the two blocks end on each other’s original mats after three moves through the empty mat, the button is pressed after each move, robot back at origin; 700), and Stack Blocks (“Stack the three blocks with different textures”; all three blocks stacked, robot back at origin; 550). Partial credit that RoboDojo awards, for the sleeves alone or for two of three blocks, is not success. Seeds.Our seedkis layoutkof official dataset seed 0. Seeds 0–4, the same five layouts as RoboDawn’s released run [14], are the evaluation seeds of Table 1. Seeds 5–9 were run separately under the same protocol with the same frozen tools; seeds 5 and 6 also served to diagnose and verify the swap-blocks environment fix described below. Execution agents. Four frozen models act as the execution agent inside their usual coding-agent harnesses: Claude Fable 5.1 (claude-fable-5-1) and Claude Opus 5.5 (claude-opus-5-5) in Claude Code 2.1.281–283 at effortxhigh, restricted to their shell and file-reading tools; GPT-6 Astra (gpt-6-astra) in the Codex CLI 0.154.0 at medium reasoning effort; and DeepSeek-V4-Flash (deepseek-v4-flash) in the open-source pi coding agent 0.73.1 with thinking athigh. Each episode runs in a fresh, empty working directory with project memory, hooks, and sub-agents disabled, from a brief that states the task instruction, the public success criterion, the step budget, and the command syntax; nothing about other episodes, seeds, or the success checker is available. The agent reaches the simulator only through one command-line client over ssh, with its own port and container, and every command is logged. Observations and depth probe. An observation returns the 640×480 head-camera image, which looks down on the table from behind the robot, the two wrist-camera images, both arms’ state, and the remaining budget; a depth probe returns the world coordinates of a head-camera pixel from public depth. Agents copy the images to their working directory and view them themselves. The same observation and probe commands are available in every condition. Every played simulator step counts against the budget; observations, probes, dry runs, refusals, and failed plans are free. Conditions.Under URAI, the agent calls the tools of the collection one at a time as described in Appendix B, choosing pixels from a named observation and the remaining arguments itself, and decides the next call after reading each reply. Under native control, the agent moves one arm’s fingertip centre to a world position, with the tool pointing down at a chosen yaw or at any end-effector orientation given as a quaternion; moves both arms together, with both tools pointing down; or sets a gripper. The same planner refuses infeasible targets at no step cost. The native brief shows the tool-down form, gives the rig facts (table height, arm bases, yaw convention, grasp height), and asks the agent to read the control script’s help, which documents the quaternion option; pouring is therefore possible natively. In the program condition, run with Claude Opus 5.5 and GPT-6 Astra on seeds 0–4, an episode has two phases. In the planning session, the agent may take observations, view the images, probe depth, read tool schemas, and dry-run tools, but may not move the robot; it then writes one Python program, using only the standard library and NumPy, that calls the same tools through the same client, takes its own observations, and branches on tool replies, the step budget, and depth probes. After the session has ended, the framework runs the program unchanged, without the model; the program may not call models or other programs besides the client, make network requests of its own, or read files outside its folder, and it ends by returning both arms home once. Stopping, limits, and continuation. The agent stops when it judges the task complete or impossible, when the remaining budget cannot cover another motion, or after 100 commands, then returns both arms home once, which records RoboDojo’s terminal state; onlysucceededcounts. In the program condition, the 100-command limit applies to the planning session, and the program applies the other stopping conditions before returning the arms home. Wall-clock limits are 45 minutes for URAI, 45 minutes for both phases of the program condition together, with at most 30 minutes for the program run, and 60 minutes for the native condition. If a program does not return the arms home, the framework does so before reading the terminal state; a program that is stopped at its limit or crashes counts as a failure. An episode 17 whose model stream ended abnormally was resumed in the same session at most twice, every agent-side failure counts as a failure, and an episode stopped at its wall-clock limit is scored by the terminal state it had reached. Timing and token accounting. Time is the wall clock of the agent’s session from launch to exit; in the program condition it spans the planning session and the program run. Model calls, output tokens (reasoning included where the harness reports it), input tokens, and viewed images are read from each harness’s own session log; Codex counts a poll of a long-running command as a model call, so call counts are not comparable across harnesses. Means are taken over all episodes, failures included, and all costs are those of the execution agent within evaluation episodes. In the timing decomposition of Table 2, motion time includes returning home; seconds per call is pooled motion-command seconds, agent-issued home commands excluded, divided by the motion-call count; percentage columns average per-episode shares, so a ratio of mean times need not equal the mean share; and Other, the time outside logged commands, includes model inference. Codex’s polling of long commands locates their end within one polling interval. Infrastructure failures. An episode is voided and rerun with the same agent, condition, task, and seed only when an infrastructure failure unrelated to the agent occurs, for example when the simulator container is stopped by another process on the shared host or the host disk is full before any motion. Agent-side failures, including timeouts and abnormal model streams that exhaust their resumptions, are never voided. Command audit. After each episode, every logged command is checked against the permitted command forms of its condition and the privileged-information rules; in the program condition, the program file and every command the program issued are checked as well. The checks cover commands outside the permitted form, low-level simulator tools called by name, hosts or ports other than the episode’s own, motion after returning home and reading the result, real motion during the planning session of the program condition, and reads of the checker source, ledgers, code trees, or other episodes. Flags are reviewed against the command and its output, and an episode with a confirmed violation counts as a failure. Isolation of evaluation episodes. Every evaluation episode starts a new agent session in a fresh working directory. Claude Code runs with automatic memory and web tools off, hooks and MCP servers disabled, and only its Bash and Read tools enabled; pi runs with--no-context-files --no-skills --no-extensionsand a dedicated empty configuration directory; Codex runs non-interactively throughcodex exec; the global instruction files that Claude Code and Codex load contain no task information. Beyond the task section, the brief gives only the episode’s own simulator port and commands; it gives no success-checker source, ledger, development notes, or paths of other episodes. We audited every command of every scored RoboDojo episode, including the commands issued by the programs of the program condition: none accessed memory, a ledger, the checker source, ground-truth keypoints, or files of another episode. Swap-blocks environment fix. On reset, RoboDojo’s articulation code rewrites the spring button’s drive target in the scene description but does not push it to the physics engine, so the button rests at 79% of its travel, and the checker’s condition that it returns above 90% after each press can never hold; official success on swap blocks is then impossible under every condition. The backend’s environment bootstrap therefore re-applies the declared drive targets after reset for any articulated object, without reference to task, seed, layout, or button position. The change ships with unit tests, touches neither the tools nor the scoring, and was verified on development seeds 5 and 6. Every condition uses the same patched environment, and swap-blocks episodes run before the fix were voided and rerun with it. Tool collection and freezing.The same thirteen reusable tools are exposed in every task under URAI and in the program condition: pour, pick-and-place, place-in, place-on-pad, push-button, push-object, sweep, swap, insert, intercept, stack- ring, stow, and strike. They were frozen before evaluation in four code trees: pour liquid and stack blocks use the original frozen tree; pour balls uses the original tree with its pour tool replaced by the task-specific pouring tool, frozen before its URAI and program episodes were rerun; swap blocks uses the original tree with the reset fix above as the only difference; and fold clothes uses its own frozen tree, whose pick-and-place tool is the revised version of Section 4.3. Each episode runs on a copy of its task’s tree, whose revision is recorded with the episode, and the execution agent may neither read nor modify the tool code. 18 C.2 AgileX real robot Physical platform. The real-robot experiments use two AgileX PiPER-X arms (6-DoF, parallel-jaw grippers with a 70 m stroke) mounted side by side on a shared base plate at the rear edge of a tabletop, with their bases about 0.59 m apart, so that both arms reach a common workspace (Figure 1). Perception uses four Intel RealSense D435 RGB-D cameras: one on each wrist (eye-in-hand) and two fixed scene cameras, one on the central mast looking down on the table and one facing the workspace from the front. The wrist cameras are calibrated eye-in-hand and the overhead camera eye-to-hand against a planar AprilTag board, and the two arm bases are registered to a common frame through the same board; the table plane is fitted from depth at run time and rejected when its residual or tilt exceeds fixed thresholds. Each arm is driven over its own CAN bus at 1 Mbit/s in joint-position mode; the backend streams joint references from the timed Cartesian path (Section 3) and closes the gripper until contact under an effort limit rather than to a fixed width. Perception models run on off-board GPU servers and foundation models are reached through their APIs; the robot host runs only capture, geometry, and control. The scene in Figure 1 shows the props used for these tasks: colored letter and number blocks, cups and a bottle, a cloth, a power strip with push buttons, plates, and a bowl. Task set. Seven tabletop tasks are run on this platform, all with GPT-6 Astra as the execution agent calling URAI tools: unscrew bottle cap, tic-tac-toe, block into bowl, hot-dog serving, tossing blocks into a bowl, pouring blocks, and clothes folding; Figure 4 shows the initial scenes. The first two correspond to GPT-Policy’s stationary dual-arm tasks R3 (unscrew bottle cap, with robot video and action context in its best condition) and R8 (tic-tac-toe, human interaction) [8]. Block into bowl repeats Robocurve’s task, picking up a block from the table and placing it inside a bowl [35]. The remaining four tasks have no published same-task reference; clothes folding is the physical counterpart of the simulated task of Section 4.3. GPT-Policy’s other stationary tasks and its Movable Exploration task, which requires a mobile base, are not ported. Unscrew bottle cap Hot-dog serving Toss blocks into bowl Fold clothes Pour blocks Figure 4. Initial scenes of the AgileX dual-arm tasks, seen from the front scene camera and cropped to a common scale: unscrew bottle cap, hot-dog serving, toss blocks into bowl, fold clothes, and pour blocks. Tic-tac-toe and block into bowl are not pictured. Backgrounds are blurred for clarity; the table, arms, and objects are unmodified. Task settings and success criteria. In unscrew bottle cap, the robot unscrews the cap from a bottle, following R3. In tic-tac-toe, the robot plays against a human opponent, moving first in the first and third games and second in the second, and a game succeeds if the robot wins or draws, R8’s criterion. In block into bowl, the robot places one block, white, red, or green in the three trials, inside the bowl. In hot-dog serving, the robot distributes two buns and two sausages so that each of two plates receives one bun and one sausage; progress is the number of items on the plates out of four. In toss blocks into bowl, six blocks lie scattered on the table and all six must end up in the bowl; progress is the number of blocks in the bowl out of six. In pour blocks, the robot pours the blocks held in a cup into a blue container and returns both arms to the home pose. In clothes folding, the robot folds a shirt on the table, and progress is the operator’s completion score. In every task, success requires the whole task to be completed. 19 Trials and speed. The three tasks with a published reference, unscrew bottle cap, tic-tac-toe, and block into bowl, are run for three trials each at1×robot speed. The other four tasks are run for ten trials each; success and progress are computed over all ten trials, and time and tokens over the first three, the only ones with a token log. The logged pour-blocks trials use a robot speed scale of 0.8. Timing and token accounting.The episode clock starts when the task is released to the execution agent and stops at a confirmed terminal outcome or the common limit. Observation, model inference, network waiting, physical motion, retries, recovery, verification, and, in tic-tac-toe, waiting for the human’s moves are included; video acceleration and nominal trajectory duration do not replace this wall-clock measurement. Tokens are execution-agent output tokens with reasoning included. Model calls count one per execution-agent model response within the token-accounting window, including model-initiated observations, polling, status queries, retries, recovery, and replies; loops inside tools are not counted. Token totals include all actual model calls, including repeated context and auxiliary calls, without double-counting cached or reasoning subsets already included in provider usage; missing usage is reported as missing rather than zero. Means are taken over all trials, failures included, and all costs are those of the execution agent. In the first bottle-cap trial, counting starts after a diagnosis-and-repair phase, once the repair was loaded; in block into bowl, the first trial includes generating an execution script that the second and third trials reuse. Protocol differences from published references.GPT-Policy runs its tasks on its own hardware, and its result table does not identify the robot used for each trial. It reports means over three trials and counts decisions, the model’s robot-tool requests, which is narrower than our model calls, which also include status queries and replies; it publishes no per-task token counts for R3 or R8. Our R3 counterpart is compared with GPT-Policy’s best condition, which gives the same model robot video and action references as context, and our tic-tac-toe keeps R8’s human-interaction semantics and success criterion. Robocurve runs GPT-6 Astra on YAM arms through absolute end-effector pose commands, with three camera views, proprioception, medium reasoning effort, a 25% robot speed cap, and a cap of 20 model calls per run, over 20 trials per task; it publishes per-run output tokens with reasoning included, and its puzzle-insertion task was not run here. Both are therefore published references from other hardware and operating limits, not same-robot controls. 20