Paper deep dive

What Stops Recursive Self-Improvement in Robotics? Lessons from 123 Rounds of Agentic Skill Discovery

Jiaming Wang

Year: 2026Venue: arXiv preprintArea: cs.ROType: PreprintEmbeddings: 98

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%

Last extracted: 9/29/2026, 11:37:32 AM

Summary

This paper reports on a 123-round experiment of an agentic system attempting recursive self-improvement in robotics for household manipulation tasks. While the agent successfully discovered and deployed new capabilities (like active viewing), the target task (placing condiments on a fridge top shelf) never succeeded. The study identifies three main bottlenecks: 1) Chained perception modules (like SAM 3) cannot understand relational concepts (e.g., 'top shelf'), leading to ineffective geometric rule accumulation; 2) Skill chains lock learning onto early failures, preventing improvement in later steps; and 3) The evaluation harness dictates what the agent learns, with weak tests and misleading memory causing stagnation. The agent itself was rarely the bottleneck.

Entities (10)

FRIDGE Task · task · 99%Agentic Self-Improvement System · system · 98%SAM 3 · model · 95%Semantic Displacement · concept · 95%Robocasa · environment · 92%Florence-2 · model · 90%PandaOmron · robot · 90%Vision-Language Model · model-class · 90%GPT 5.6 Sol · model · 85%GPT-6 Astra · model · 85%

Relation Signals (10)

Agentic Self-Improvement System → attemptedtosolve → FRIDGE Task

confidence 99% · We pointed it at a household task in RoboCasa [7], placing condiments on the top shelf of a fridge, and let it run for 123 improvement rounds

SAM 3 → lackscapability → relational_understanding

confidence 95% · SAM 3 is built for concept prompts... and has no notion of which one is 'top' or 'first'.

Agentic Self-Improvement System → sufferedfrom → Semantic Displacement

confidence 95% · The agent, which can only write code, filled that gap with ever more geometric rules that never converged... We call this semantic displacement

Agentic Self-Improvement System → usedmodel → SAM 3

confidence 95% · Segmenters such as SAM 3 find shelves but not 'the top shelf'

Agentic Self-Improvement System → operatedin → Robocasa

confidence 92% · We pointed it at a household task in RoboCasa [7]

Agentic Self-Improvement System → controlled → PandaOmron

confidence 90% · We used three RoboCasa [7] kitchen tasks with a PandaOmron mobile manipulator.

Agentic Self-Improvement System → needed → Vision-Language Model

confidence 90% · What was needed was a different kind of model, a vision-language model

Agentic Self-Improvement System → →

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Can a robot improve itself the way coding agents now improve software? We built an agentic system to find out. It watches a robot fail, works out which capability is missing, writes new skills or finds and installs external models, tests every change in simulation, and repeats, with no human writing robot code. We ran it for 123 improvement rounds on household manipulation tasks. This report describes what we learned. The good news is that the agent can discover capabilities on its own: noticing that its targets were out of view, it asked for an active-viewing model, debugged it, and deployed a working search skill. The bad news is that its improvements did not add up. Changes kept passing their tests, yet the target task, putting condiments on the top shelf of a fridge, never succeeded. We found that the agent was rarely the bottleneck. Three things around it were. First, chained perception modules do not understand relations. Segmenters such as SAM 3 find shelves but not "the top shelf", so the agent filled the gap with ever more geometric rules that never converged, when what it needed was a different kind of model. Second, skill chains lock learning onto the first step. Long tasks mostly fail early, so evidence and fixes pile up there, and later skills are rarely reached, tested, or improved. Third, what the agent learns is decided by the harness. The agent optimized exactly what the evaluator measured, including where it was wrong, and weak tests and misleading memory turned activity into a standstill. We distill these lessons into concrete recommendations for building robot systems that improve themselves, each paired with an experiment that could prove it wrong.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

95,055 characters extracted from source content.

Expand or collapse full text

What Stops Recursive Self-Improvement in Robotics? Lessons from 123 Rounds of Agentic Skill Discovery Wang Jiaming National University of Singapore Technical report Abstract. Can a robot improve itself the way coding agents now improve software? We built an agentic system to find out. It watches a robot fail, works out which capability is missing, writes new skills or finds and installs external models, tests every change in simulation, and repeats, with no human writing robot code. We ran it for 123 improvement rounds on household manipulation tasks. This report describes what we learned. The good news is that the agent can discover capabilities on its own: noticing that its targets were out of view, it asked for an active-viewing model, debugged it, and deployed a working search skill. The bad news is that its improvements did not add up. Changes kept passing their tests, yet the target task, putting condiments on the top shelf of a fridge, never succeeded. We found that the agent was rarely the bottleneck. Three things around it were. First, chained perception modules do not understand relations. Segmenters such as SAM 3 find shelves but not “the top shelf”, so the agent filled the gap with ever more geometric rules that never converged, when what it needed was a different kind of model. Second, skill chains lock learning onto the first step. Long tasks mostly fail early, so evidence and fixes pile up there, and later skills are rarely reached, tested, or improved. Third, what the agent learns is decided by the harness. The agent optimized exactly what the evaluator measured, including where it was wrong, and weak tests and misleading memory turned activity into a standstill. We distill these lessons into concrete recommendations for building robot systems that improve themselves, each paired with an experiment that could prove it wrong. 1 Introduction We want robots that get better on their own. Today, when a robot fails at a household task, a person watches the video, works out what went wrong, and writes new code or collects new data. Coding agents suggest a different route. They can already read logs, form hypotheses, write and test code, and keep going for hours. Such agents already improve systems in loops: Voyager builds a growing library of Minecraft skills [1], Eureka evolves reward functions for robot learning [2], and the Darwin Gödel Machine rewrites its own agent code [3]. The idea has now reached robots: RHO optimizes robot-policy code repositories [4], ASPIRE grows a library of robot skills from diagnosed failures [5], and ENPIRE lets coding agents improve real-world robot policies [6]. So the idea is appealing. Let an agent watch the robot fail, figure out which capability is missing, write it or find a model that provides it, test it, and repeat until the task works, with no human writing robot code. We built such a system and tested this idea (§2). Coding agents write the task programs, improve a library of robot skills, and research and install external perception and planning models. Every change must pass physical tests in simulation before the robot uses it. Humans were allowed to fix only the testing harness, never a skill. We pointed it at a household task in RoboCasa [7], placing condiments on the top shelf of a fridge, and let it run for 123 improvement rounds, several weeks of continuous operation. We went in with three questions: 1. Can a coding agent discover a missing robot capability from failures alone? 2.Do the improvements it makes add up, so that each round starts from a better robot? 3. If not, what stops them, and is it something a stronger agent would fix? The answer to the first question is yes, sometimes. The agent noticed that its targets were out of view, asked for an active-viewing model, debugged it, and deployed a search skill with no human robot code (§3). The answer to the second question is no. The system kept producing improvements that passed their tests, yet the fridge task never succeeded, and in the last twenty rounds the robot failed at the same step every time. Most of this report is about the third question. We found that the agent was rarely the bottleneck. What stopped progress was what surrounded it. We organize what we learned around one observation (§4): an improvement helps only if the failure is described correctly to the agent, the fix is within what the agent can change, and the test can tell whether it helped. In software all three come almost for free; in robotics each one broke, in a way that taught us something: • Lesson 1: chained perception modules do not understand relations (§5). Household instructions say things like “the top shelf”. SAM 3 can find shelves, but not which shelf. Chaining it with other segmenters and depth does not help, because the relation lives between the modules. The agent, which can only write code, filled that gap with ever more geometric rules that never converged. What was needed was a different kind of model, a vision-language model, and switching model class was not an action the agent could take. •Lesson 2: skill chains lock learning onto the first step (§6). A fifteen-step task mostly fails early, so almost all evidence and almost all fixes concentrate on the first step. Later skills are rarely reached, rarely tested, and their improvements never reach the task. •Lesson 3: what the agent learns is decided by the har- ness (§7). The agent optimized exactly what the evaluator 1 arXiv:2609.31760v1 [cs.RO] 23 Sep 2026 measured, including where the evaluator was wrong. Weak tests, an unchanging stream of evidence and a memory that recorded untried ideas as failures turned activity into a stand- still. Keeping this harness honest took about two human repairs for every change the agent got accepted. We close with what we would do differently (§8) and how to test each suggestion. This is a report of one long experiment, not a controlled study (§10). We think it is useful precisely because it failed in instructive ways that anyone building self- improving robots is likely to meet. 2 System overview The idea is simple: treat the robot’s software as something an agent can rewrite, and let failures tell it what to rewrite. Figure 2 shows the system. The robot is controlled by a task program that calls skills. Skills call external models. Every part except the evaluation harness can be changed by an agent or by learning, without a human writing robot code. The robot’s software. A task program is Python code that composes skills and declares what each step should achieve, for example “the top shelf inside the fridge is localized”, “the object is grasped and lifted”, or “the object is placed”. A skill (localize an object, find a region, approach, grasp, move the arm) has typed inputs and outputs and can have several im- plementations. Routing rules choose among them. Skills can call providers: external models behind a contract, such as text- prompted segmentation (SAM 3 [8]), referring-expression seg- mentation (Florence-2 [9]), monocular depth, 6-DoF grasp gen- eration or motion planning. The robot perceives only through its head and wrist RGB-D cameras and its joint readings, as a real robot would. How each part improves. Programs are written by an or- chestration agent from the task description. It keeps its best past program and revises it only for program faults, so that skill failures are fixed in skills rather than worked around. Skills are improved by a proposer agent. For each selected gap, it sees the recorded failures, the evaluator’s diagnosis and its own earlier attempts, and writes a code change. Providers are added by a research agent. When the proposer decides that no code change can fix a failure, for example because a model is miss- ing, it files a request. The research agent then finds a model, installs it, checks it on a probe scene and on recorded failing inputs, and registers it. Routing is refitted every round from logged skill calls. The harness, meaning the evaluators, tests, memory and the description of the robot’s actuators, is the one part only humans may change. Our rule was that humans can repair the harness but never write a skill, program or model wrapper. How a change is accepted. The evaluator judges each de- clared step with privileged simulator state and returns a short diagnosis, such aswrong_region,no_grasp_contactor tool_path_blocked. A proposed code change must pass a Table 1: Agent models. All agents used the same model and reason- ing setting in a given round. Rounds 1–75 gpt-5.6-sol, maximum reasoning effort Rounds 76–84 gpt-6-astra, low reasoning effort Rounds 85–123 gpt-6-astra, medium reasoning effort InterfaceCodex CLI, one session per agent task, in a read- only, capability-dropped container; from round 74 the run repository and held-out task are not visible OrchestratorTask description, skill catalog with observed usage, its best earlier programs; up to 25 min per program ProposerFailure table, evaluator diagnoses, execution traces and diffs of earlier attempts, camera keyframes of each failed step; up to 3 attempts per problem, 60 min each Research agent The request and recorded failing inputs; finds, in- stalls, wraps and tests an external model sequence of physical tests. First, it replays the exact recorded failure with the change swapped in; did it fix what it claims to fix? Second, it runs paired trials, old against new, on fresh scenes and on a held-out task the proposer never sees; does it help without hurting elsewhere? Later in the project we added tests on sealed recorded failures and on the other skills a change touches. Accepted changes go live in the next round. The agent models. All three agents (orchestrator, proposer and research agent) are OpenAI GPT models run through the Codex CLI, each session in its own sandboxed container (Ta- ble 1). The proposer is multimodal. Along with the failure table, the evaluator’s diagnosis and its earlier attempts, it re- ceives RGB keyframes from the robot’s cameras at the start and end of every failed step, attached as images. It can there- fore look at the failures it is asked to fix. The model and its reasoning effort changed twice during the run, both times by our decision, not the agent’s: fromgpt-5.6-solat maximum reasoning effort togpt-6-astraat low effort (round 76), for speed, and then to medium effort (round 85). The robot itself never calls these models. Its perception comes only from the providers its skills call. Tasks. We used three RoboCasa [7] kitchen tasks with a PandaOmron mobile manipulator. The target task was FRIDGE: “Place the mustard and mayonnaise from the counter to the top shelf of the fridge. If the existing items in the fridge are on the top shelf, move them to other shelves.” The condiments start out of view, and the task takes roughly fifteen steps. Two simpler pick-and-place tasks, CABINET (counter to cabinet) and SINK (counter to sink), were added later so that skills had to generalize across tasks. A fourth task, cabinet to counter, was held out for testing only. A round took one to three hours, plus several hours when models were researched, and we ran 123 rounds. 2 255075100 Round 0 20 40 60 80 Cumulative count 57 86 (a) Changes keep accumulating Accepted skill changes Registered models 255075100 Round 0 1 2 Successes (of 4 per task) 14 sink successes 0 fridge / 0 cabinet (b) Task successes per round Fridge: 0 throughout 0102030 Accepted changes Grasp Approach Find region Find object 3 3 20 31 (c) What the changes touched Figure 1: A busy loop that did not solve its task. (a) Accepted skill changes and newly registered models keep accumulating over 123 rounds. (b) Task successes per round: none on the target FRIDGE task; the few on SINK were mostly accidental. (c) Almost all accepted changes are to perception, the first step of every task (Lesson 2). 1 Collect run task programs in simulation 2 Judge evaluator checks each declared step 3 Diagnose group first failures into gaps 4 Propose agent edits skills or requests a new model 5 Test & accept replay failure, fresh states, held-out task Research install, probe, register an external model code request next round starts from the improved robot Task program composes skills and declares step goals Routing chooses among a skill’s implementations Harness evaluators, tests, memory, robot interface description Models (providers) segmenters, VLMs, grasp generators, motion planners Skill library localize, find region, approach, grasp, move orchestrator agent keeps or revises fitted from logged calls humans only register promote Figure 2: How the system improves itself. Each improvement round (top row) runs the robot, judges every declared step, groups failures, and lets a coding agent (a GPT model in the Codex CLI, shown camera keyframes of each failure) propose a fix, either as code that must pass physical tests or as a request for a new external model. Five parts of the robot can change (bottom row); arrows show who changes each one. Four are changed by agents or by learning from logs. The harness that judges and remembers is changed only by humans, which turns out to matter (§7). 3 What happened over 123 rounds Figure 1 summarizes the whole run, and Figure 3 shows, round by round, where each task first failed. The story has three phases. Phase 1: discovery (rounds 1–36). Early on, the loop did what we hoped. The condiments in FRIDGE start out of view, and nothing in any prompt said so. At first the localizer re- turned confident points on visible look-alikes, and the agent tried to re-rank them. Once we fixed the evaluator so that it reported “the target is not visible from any camera”, the agent changed approach: it asked for a “target-free, collision-aware active viewpoint selection” model, found a bug in the planner it was given (the robot’s own start cell was marked as occupied), requested a repair, and wrapped the result in a look-around skill. By round 16 the robot searched when it could not see its target, and out-of-view failures disappeared. In the same phase it created a new skill family for finding regions, asked for a referring-expression segmenter, a grasp generator and motion planners, and, once the region evaluator was made truthful, produced a top-shelf fix that passed every test. Phase 2: the manipulation front and the harness (rounds 37–102). With perception moving, failures shifted to reach- ing and grasping, and progress there exposed the harness rather than the agent. The reach evaluator turned out to be a sphere, and the agent had learned to park exactly on its surface. The robot’s torso lift was missing from its action description. The base controller behaved very differently from its documenta- tion. Each fix opened a new front. The first approach skill was accepted in round 62 and the first grasp skill in round 63; the CABINET and SINK tasks were added so that skills had to work across tasks. The first task successes, on SINK, came in round 78. They did not hold: in later rounds the same program mostly succeeded by accident, when a released object hap- pened to fall into the sink. Meanwhile the top-shelf problem came back as programs and viewpoints drifted. By the end of this phase, the fridge and cabinet rollouts were identical from one round to the next. Phase 3: more capability, same outcome (rounds 103–123). We repaired the loop’s memory and tests: lost research requests were recovered, partial fixes were allowed to reach the fresh- scene tests, and small differences had to reproduce before they 3 66768594103111118123 Discovery round (one task/seed per row) Fridge 68 Fridge 69 Fridge 70 Fridge 71 Cabinet 68 Cabinet 69 Cabinet 70 Cabinet 71 Sink 68 Sink 69 Sink 70 Sink 71 Region Object Retain Reach Position Task All effects pass Error/no trace Figure 3: Where each rollout first failed, rounds 66–123. Each row is one task and scene; color is the first step that failed. White circles mark task successes. Early in this window failures move around as skills change; in the last twenty rounds the fridge rows freeze at region finding (“the top shelf”) and grasping, and stay there despite continued library changes and new models. counted. The research agent then registered new models every round, including the first vision-language grounders for the top-shelf problem. A grasp improvement in round 118 won reproduced comparisons on unseen scenes, including the held- out task. None of it moved the target task. In every round of this phase, three of the four fridge scenes failed at finding the top shelf and the fourth at grasping. The new grounders were never called by the robot. What worked. •The agent can discover a missing capability when the failure is described correctly. The active-search skill was invented, researched, debugged and deployed with no human robot code. • The agent can acquire and adapt external models. Eight agent-introduced models (grasp generation, motion and base planning, active viewing, relational localization) were used in deployment. •Carrying partial designs forward works. Designs con- verged once rejected attempts returned their execution traces and the best earlier design seeded the next attempt, instead of each attempt starting over. •The agent is a good diagnostician. When it declined to change a skill and explained why, it was often pointing at a real harness bug. What did not work. • The target task never succeeded. Neither did the cabinet task; the sink successes were mostly accidental drops. •Changes piled up at the front of the task. Almost all accepted changes were to perception, the first step of every task (Fig. 1c). •Local improvements did not add up. Changes that passed rigorous tests, and new models that passed research, did not change what the robot achieved. •Keeping the loop honest took constant human work, about two harness repairs for every accepted skill change. The rest of this report explains why, and what we would change. 4 Why improvements did not add up A minimal model of the loop. LetΦ t be the robot’s library after roundtand let a task be a chain ofmdeclared steps. Writep j (Φ)for the probability that stepjsucceeds given that steps1,..., j− 1did, andπ j = ∏ i< j p i for the probability that a rollout reaches stepj. Task success isS(Φ) = ∏ m j=1 p j , which requires no independence assumption. Each round, the loop observes failures, a proposer produces a changeδ, and a gateGaccepts it, so thatΦ t+1 = Φ t ⊕ δwhenG(δ) = 1. Self-improvement compounds when accepted changes raiseS in expectation, so that each round starts from a better robot than the last. Three conditions. Whether an accepted change raisesSde- pends on three conditions. None of them concerns the pro- poser’s coding ability. 1.Representability. The failure must appear in the evidence the proposer sees, attributed to its actual cause. If a grasp fails because the arm stopped short, and the evidence reads “no grasp contact”, the proposer will improve the wrong thing, and do it well. 2.Expressibility. A fix must exist in the substrate the proposer is allowed to change. A proposer that can edit code but 4 (a) What the loop built: modules chained by glue “top shelf in the fridge” concept / referring seg. depth lift per mask geometric glue (33 thresholds) what a shelf is, not which relation decided by fitted rules (b) Reason first, then verify “top shelf in the fridge” VLM grounds points / boxes segment + depth lift geometric veto; abstain →look relation resolved over the scene geometry checks, does not decide Figure 4: Where the relation lives. (a) With concept-level and referring-expression segmenters, no module represents “top” or “be- low”; the relation is implemented in agent-written glue. (b) A vision- language model resolves the relation over the whole image; geometry and active viewing only verify or reject its answer. cannot change what a perception model understands can only approximate missing semantics with code. 3.Resolvability. The gate must be able to tell a better change from a worse one on the states where the change matters. A gate that sees too few relevant states, or noise larger than the effect, selects at random or by accident. In software self-improvement the three hold almost for free. A failing unit test states its cause, the fix lies in the code the agent edits, and the test is exact and cheap to re-run. In a robot loop each condition has to be engineered, and each broke in our experiment in a characteristic way. Reading our run through the three conditions. Table 2 places the main problems of the run on these axes. The one front where all three conditions held, active search, is the one the agent solved on its own (§3). Its early failure was a representability failure: while the evaluator blamed the wrong cause, the agent kept improving the wrong thing. This tells us that the proposer was not the bottleneck. The next three sections are our three lessons about what is. 5Lesson 1: chained perception mod- ules do not understand relations The fridge instruction reads: place the condiments on the top shelf; if items already occupy the top shelf, move them to other shelves. Every program the orchestrator wrote therefore asks perception for regions defined by relations: “the top shelf inside the fridge”, “the shelf immediately below the top shelf”, “a food item on the top shelf”. This is exactly the kind of query household instructions are made of, and the loop never learned to answer it. Region grounding became the most common first failure on both the fridge and the cabinet task. In the last twenty rounds, three of the four fridge seeds failed at it every round, typically by returning the shelf below the top shelf, sometimes with high confidence. This section explains why a capable code-writing agent could not fix this, and why the way it tried is itself the problem. Modules do not compose into relations.The agent had two grounding models to build on: SAM 3 [8], prompted with text, and a Florence-2 referring-expression segmenter [9] that the agent itself requested in the first round. Both are strong models. Neither represents the part of the query that matters. SAM 3 is built for concept prompts, i.e. short noun phrases. It finds shelves, all of them, and has no notion of which one is “top” or “first”. Florence-2 accepts the full expression, but for relational queries it returned diffuse masks spanning several shelves. Chaining the two with depth does not help, because depth adds metric position without adding meaning. The relational part of the query (ordinal position, “below”, “inside”, “occupied by”) is therefore represented by none of the modules in the chain, and must be resolved, if at all, by the code between them (Fig. 4a). Semantic displacement. This is where a code-writing op- timizer does something predictable and harmful. It cannot change what a model understands, so it implements the miss- ing semantics in the substrate it can change: geometry. The fridge region skill accumulated rules about recesses, occluding boundaries, adjacent thin regions and scene-scale edges, each with a tunable threshold, until it held 33 thresholds in about three thousand lines, composing an object localizer with 48 more. We call this semantic displacement: a semantic capabil- ity that the substrate lacks gets displaced into procedural code that approximates it on the states where the loop sees it fail. Displacement has a recognizable signature, and every part of it appeared in our run: •It fits states, not meaning. The fridge region gap had two recorded states. Across 27 attempts, one seed was rescued every time and the other never. Rules that encode “the top shelf” for one viewpoint do not encode it for another. •It passes gates and then decays. A top-shelf fix passed every gate in round 36 and deployed well, and the front reopened once programs and viewpoints drifted. Tests on the states where the rules were fit cannot detect this. •It grows monotonically. Each new failure adds a rule; no failure removes one. Accepted changes concentrated in per- ception (Fig. 1c) partly because perception is the first link of every chain (§6), and partly because displaced semantics keep generating new failures to patch. Displacement is not an agent error. Given the substrate, it is the best available move, and a stronger proposer would make it faster. Seeing versus understanding.The problem is made harder because relational perception is entangled with viewpoint. With the fridge door open and the robot beside it, the top shelf’s own structure is invisible; only the items standing on it show, sometimes as a few hundred pixels and on one seed as almost nothing. A system must therefore tell apart “I cannot see the region”, which calls for moving, from “I cannot tell which region”, which calls for better reasoning. Displaced geometry cannot make this distinction: a rule tuned to answer 5 Table 2: The main problems of our run, through the three conditions.✓holds,✗fails,◦holds only after harness repair. The one front where all three held is the one the agent solved on its own. FrontRepr.Expr.Resol.What happened Target outside every view (active search) ◦✓Once the evaluator reported visibility, the agent requested an active-viewing model, repaired it, wrapped it in a look-around skill, and the failure disappeared from deployment (§3). “The top shelf” (relational re- gion) ◦✗No module represents the relation; the agent encodes it as geometry fit to two recorded states. The skill grows without converging (§5). Grasp retention✗✓✗ Motion faults reported as grasp faults; fresh tests rarely reach the grasp. Real local wins never reach the task (§6). Reachability✗✓ A spherical evaluator and a misdescribed actuator; the agent optimizes the evaluator exactly (§7). Requested models✗✓✗Requests silently dropped and remembered as failed; registered models never called (§7). “top shelf” will answer something whether or not the shelf is visible. The evaluator had the same difficulty. For much of the run it could not tell these cases apart either, and early on its own language matching failed on “refrigerator” versusfridge and accepted a “top shelf” answer that landed on the lower shelf’s front lip. Open-vocabulary semantics are hard for the judge as well as for the robot. The escape is a change of model class, and it lay outside the loop.A 7B open-weights vision-language model (Qwen2.5- VL [10]), queried directly on recorded head-camera frames, placed “the top shelf” and “the shelf below” correctly in all four recorded head views we tried, in about a second per query. Models of this class resolve relations over the whole scene rather than per mask [11,12]. The loop did not get there on its own. Its proposers kept asking for repairs of the Florence-2 wrapper (“more region specificity”, “more hypothesis diver- sity”), which is how an agent reasons when its task is to fix a failing component rather than to question the component’s class. Many of those requests were silently lost (§7). There is an irony here. The proposer is itself a large vision-language model, and it was shown camera keyframes of every failed top-shelf step (§2). Models of its kind can tell which shelf is on top in such an image. What it could not do was give the robot that ability, because the only channel from the agent’s understanding to the robot’s behavior was code. The agent’s own perception did not transfer to the robot’s perception. The vision-language class entered only after a human suggested it. Even then, the first agent-built grounders that passed research were registered under a family whose code never calls them, and so never ran on the robot. Takeaway 1. Open-instruction perception fails in a self-improving loop not because each module is weak but because the relation lives between modules, where only code can reach it. A code-writing agent then displaces the missing semantics into ever-growing ge- ometry. The loop needs perception whose representation contains the relation (reason first, verify with geometry), and it needs model- class change to be an action the agent can take, not a repair it can request. 6Lesson 2: skill chains lock learning onto the first step A household task is not a skill but a chain of them. The fridge task needs roughly fifteen declared steps: find the top shelf, clear it, then twice find a condiment, approach, grasp, carry, find the shelf again, place and release. This section argues that chaining does more than multiply small unreliabilities. It also shapes what the loop can learn, in a way that keeps it stuck at the front of the chain. Double attenuation. Consider a change that raises the reli- ability of stepjby∆ p j and leaves everything else fixed. Its effect on task success is ∆S = π j |z reach ∆ p j ∏ i> j p i |z continue ,(1) so a gain is discounted both by the chance of reaching stepj and by the chance of finishing after it. The same reach factorπ j also discounts the evidence about stepj. Ofnrollouts or fresh test states, only aboutnπ j exercise stepjat all. Downstream steps therefore produce few failures to learn from, and their gates see few states on which a candidate can differ from the incumbent. A downstream repair is attenuated twice, once in its value and once in its measurability, by the same upstream unreliability. We call the result frontier lock: learning concen- trates on the first failing link because only there is evidence plentiful and gates resolvable, and the rest of the chain stays under-trained no matter how many rounds run. Figure 5a shows the lock directly. More than half of the fridge rollouts failed the first step, and almost none got past the fourth. The perception front absorbed nearly all accepted changes (Fig. 1c) not because the agent preferred perception but because that is where the loop’s evidence was. The grasp shows the other side. When a grasp candidate was tested on fresh states, most of those states failed at localization or reach in both arms, so the grasp never ran. Several acceptance rules (an absolute success floor, a multi-task span requirement, a holdout requirement) were at some point impossible for any 6 0246810 k (consecutive verified subgoals) 0.0 0.2 0.4 0.6 0.8 1.0 fraction with prefix ≥ k (a) How far the chain gets, rounds 66–123 Fridge (n=224) Cabinet (n=232) Sink (n=230) 05101520253035 numbered harness fixes (of 113) Replay and branching mechanics Embodiment and sensing interface Evaluator truth and attribution Provider research pipeline Runtime infrastructure Validation gates and statistics Memory, feedback, and selection 4 10 12 13 17 24 33 (b) What maintainers had to repair Figure 5: (a) Frontier lock. Fraction of rollouts in the three-task phase whose firstkdeclared steps all passed. More than half of fridge rollouts fail the first step and almost none reach the fifth, so steps beyond the frontier generate little evidence and are rarely exercised by gates. (b) Where the human effort went. The 113 harness repairs made by humans, grouped by what they repaired (the round each one was found in is listed in Table 4). Most concern what the agents were told, remembered and rewarded for, not the simulator or the replay machinery (§7). grasp candidate to satisfy for this reason alone. A grasp change that did win reproduced paired comparisons on unseen states, including one on the held-out task, was followed by five rounds without a single task success. That is Eq. 1 observed: a real local gain multiplied by near-zero reach and continuation fac- tors. Failures surface downstream of their causes. In a chain, the step that fails is often not the step that is wrong. A localizer that returns a confident wrong point causes the reach, grasp and place that follow to fail, each for its own reason. A pre- grasp motion that stops short produces an evaluator verdict of “no grasp contact”, which is literally true. A motion that cannot reach its target often means the base was parked in the wrong place two steps earlier. Because the loop attributes each failure to the step where it surfaced, it sent proposers to repair the wrong family, sometimes for many rounds. This is the representability condition failing because of composition, not because of any single evaluator. The corrective is to judge each step by what it could have done from where it started. For example, the evaluator can run its own reach trial before judging a motion, and reassign the failure to placement when even that trial cannot get there. Composition inside skills, and the chain nobody learns. Skills also call skills: region finding calls the object localizer, and the grasp calls motion and approach. A change to one can silently break another that composes it, and until late in the run the replay machinery could not branch at a nested call at all, so gaps rooted inside a composed skill were untestable. Above the skills, the chain itself was not a learning target. The orchestrator wrote each program once and was told to route every failure into the skill library, not around it. It never learned to insert a missing step, such as raising the torso and looking again before grounding a shelf. Every chain-level problem therefore reappeared as some skill’s gap, where it could not be fixed. Task success is not program success.The easiest task, plac- ing an item in the sink, is the only one that ever succeeded, and most of its successes were accidents. The program’s own place steps had failed, the gripper opened elsewhere, and the object happened to fall into the sink. The same program al- ternated between success and failure across rounds because an upstream estimate took one of a few values depending on contact physics. Native task success and correct execution of the intended chain are different measurements. A loop that counts the first as progress will reinforce luck. Takeaway 2. Chaining makes self-improvement path-dependent. Upstream unreliability discounts both the value and the measur- ability of every downstream repair, so the loop locks onto the frontier and never gathers evidence for the rest of the chain. Robot self-improvement must learn at the level of the chain: skills with explicit preconditions and recovery, attribution by what a step could have done from where it started, an orchestrator that can edit the chain itself, and test states drawn to exercise each step rather than only the first. 7Lesson 3: what the agent learns is decided by the harness In a self-improvement loop the proposer supplies variation; everything else (the evaluators that say what went wrong, the gates that accept or reject, the memory that carries experience forward) supplies selection. In software the selection side is a test suite. In robotics it is a set of fallible models of physics and meaning, written by people and never complete. This section argues that the selection side, not the proposer, determined what our loop learned. We identify four mechanisms by which it failed. Each one turns a capable optimizer into an efficient producer of the wrong thing. Evaluator capture.A strong optimizer finds the evaluator’s errors before it finds the task. For much of the run, “reachable” was judged by a sphere around the arm base. The agent’s 7 approach skill learned to park the base exactly on the sphere’s surface: hundreds of recorded successes lie within a millimeter of the advertised radius, and the grasps that followed could not reach. Meanwhile, the one physical mechanism that did make the top shelf reachable, a torso lift, was missing from the action description. Proposers reported dozens of times that they lacked “the actuator layout”. The base controller’s advertised speed was off by more than an order of magnitude and hid a dead zone in which every short move stalled. None of this is reward hacking in the adversarial sense [13]. The agent did exactly what it was told, and it was told wrong things. The deeper point is that in a robot loop the evaluator and the embodiment description are models of the world, not specifications. Their error is a second optimization target that the loop exploits by default, and the more capable the proposer, the faster it does so. Unresolvable selection.A gate is a statistical test on a hand- ful of physical rollouts, and ours failed in every direction a test can fail. It was too coarse: exact-state replay demanded a 60% rescue rate on gaps that usually had one or two recorded states, so a design that rescued one of two was rejected by arithmetic, and the one gate that measures generalization on fresh states was unreachable for partial designs. It was too permissive: when fresh baselines were already saturated, the required improvement was zero, and about two dozen changes were accepted on replay of the very states the proposer had seen, which is precisely what an over-fitted patch passes. It was noisier than the effects: when disagreeing fresh-state pairs were re-run, nearly half did not reproduce. Contact physics, upstream estimates and provider behavior vary between runs even from a shared seed. And it was blind beyond the frontier (§6). Every one of these was fixable, and was fixed. The gen- eral lesson is that resolvability must be designed: thresholds that respect the sample size, reproduction of every difference, states drawn to exercise the changed skill, and a lockbox the loop has never selected against [14, 15]. The evidence fixed point. The simulator is deterministic, collection reused four seeds per task, and the orchestrator’s memory froze its best programs. Together these made the loop’s input constant. For more than a dozen consecutive rounds the fridge and cabinet rollouts were identical, the miner re-filed the same failures, and proposers answered gaps they had seen over twenty times, reading hundreds of thousands of tokens of their own history per attempt. A loop whose obser- vations do not change cannot learn anything new from them, however good its proposer. Repetition makes a useful regres- sion sentinel; discovery needs a separate, deliberate source of novelty. Memory corruption. Memory is what makes the loop re- cursive, and a corrupted memory is worse than none. The proposer could ask for research on a missing capability. For much of the run, requests written to one of the two permitted locations were treated as library code: gated, found unable to move the robot, and recorded as failed attempts. On the fridge-region front the same request, for a more specific region model, was filed round after round and never researched. The proposer, reading its history, concluded the request was on file and declined the gap, writing “I do not duplicate that request”. An untested hypothesis had been recorded as a refuted one, and the record then suppressed the very exploration that could have escaped semantic displacement (§5). Beliefs went stale in the same way: a proposer argued from its history that a repaired model was not being served, contradicting every trace of the current round. The fix was to give memory a type system: every hypothesis is recorded as tested and rejected, untestable, failed on infrastructure, pending, or never run, and only the first counts against it. Capability is not use. The research pipeline eventually turned requests into registered models at a steady rate, yet none of the eleven models first registered in the final rounds was ever called by the deployed robot. Registration showed that a model answers a synthetic probe and does not break what was tested. It did not show that any skill routes to it, and no part of the loop was responsible for integration. Framing outweighs effort.Proposer behavior was far more sensitive to how the harness described its own gates than to how hard the proposer thought. A single paragraph warning that state-specific repairs are rejected coincided with proposers declining every gap for two rounds. A paragraph explaining that rejected attempts seed the next one coincided with nearly every attempt becoming a real edit, one of which was promoted. Raising the proposer’s reasoning effort changed nothing visible. The harness’s text is part of the learning algorithm. Who actually learned.Over the run, humans made 113 num- bered repairs to the harness (Fig. 5b), about twice the number of library changes the agent had accepted. Most corrected what the agents were told, remembered or rewarded for. Several were first diagnosed by the agent itself: a proposer that de- clined a gap and explained why was often pointing at a harness bug, such as nested calls renumbered on replay or a program asking the object finder for “cabinet interior”. In this sense the loop was recursive in the wrong place. The agent improved the robot within a harness that only humans improved, while the agent’s most accurate diagnoses concerned the harness. Takeaway 3. In robotics the selection side of self-improvement (evaluators, embodiment descriptions, gates, memory and their wording) is a model of the world, not a specification. A capable proposer exploits its errors, cannot learn past its resolution, stalls when its inputs stop changing, and is misled by its own history. Recursive self-improvement must therefore extend to the selection side, with its own tests. The agent’s reasoned refusals are a ready source of harness bug reports. 8 What we would do differently If we started again, we would build the loop differently. Below are our recommendations, grouped by which of the three con- 8 ditions each one restores. They come from this experiment but were not themselves tested in it, so Table 3 pairs each with an experiment that could prove it wrong. Restoring representability: say why, where it started. (D1) Attribute failures by capability, not by location. An eval- uator should ask whether a step could have succeeded from the state in which it began, and assign the failure to the step that made it impossible. This counters the downstream sur- facing of failures in chains. (D2) Measure the embodiment in- stead of declaring it. Actuator response, dead zones, reachable workspace and self-occlusion should be probed and published as data, and re-probed when they change. An agent can only exploit capabilities it is told about, and it will exploit errors it is told. (D3) Feed back mechanism, not verdicts. Rejected attempts should return execution traces and structured diagnos- tics. In our experiment this was the difference between designs that were re-derived from scratch and designs that converged. Restoring expressibility: widen what the agent can change. (D4) Reason first, verify with geometry. For open-vocabulary instructions, perception should include a model whose repre- sentation contains relations, such as a vision-language model producing points or boxes [10–12]. Segmentation [8,16] and depth should convert its answer into geometry, and geometric checks should veto rather than decide. Perception must be able to abstain, and abstention should trigger looking. (D5) Make model-class change an action. The loop should detect semantic displacement (growing rule counts, rescues that hold on one recorded state but not its sibling, repeated repairs of one residual) and respond by researching a different class of model rather than repairing the current one. (D6) Put the chain in the substrate. The orchestrator should be allowed to edit the chain (insert a look-again step, reorder, add recovery), and such edits should be gated like skill changes. Otherwise chain-level problems can only be misfiled as skill gaps. Restoring resolvability: test what the change could affect. (D7) Draw test states to exercise the changed step. Counter frontier lock by sampling or constructing states that reach the changed step (for example by resetting the simulator past the frontier) instead of only running the task from the start. (D8) Make gates statistical. Apply rate thresholds only where the sample can resolve them, require every paired difference to reproduce, exclude cases the change could not influence, and keep a lockbox the loop never sees. (D9) Separate sentinels from discovery. Keep a fixed regression panel, but spend a de- clared share of each round on new states, layouts or viewpoints, so the loop’s input keeps changing. Closing the recursion. (D10) Type the memory. Record every hypothesis with its test status, carry measured partial successes forward with their traces, and expire conclusions whose evaluator has changed. (D11) Require use. Admitting a model should create an integration obligation: a deployed skill must call it on relevant states before the admission counts as capability. (D12) Improve the harness in the loop. Treat evaluators, gates and harness text as learnable, with their own gate: replay the recorded archive under a proposed harness change and check which past decisions flip and why. Route the agent’s reasoned refusals to this channel, since they were the most reliable harness bug reports we had. 9 Related work Self-improving agents. Voyager grows an executable skill library with an automatic curriculum in Minecraft [1]. Eureka evolves reward code for robot learning [2]. Reflexion turns verbal feedback into better attempts [17]. The Darwin Gödel Machine keeps an archive of self-modified coding agents under empirical selection [3]. In the terms of §4, these settings largely satisfy representability, expressibility and resolvability by construction: game state, rewards, unit tests or benchmark scores are exact, cheap, and aligned with the objective. Agentic self-improvement for robots.Three recent systems are closest to ours. RHO optimizes multi-file robot-policy repositories with tool-using coding agents and execution feed- back [4]. ASPIRE lets coding agents write and refine robot control programs in an open-ended loop [5]. A closed-loop execution engine exposes fine-grained multimodal traces for diagnosis and repair, validated fixes are distilled into a growing skill library, and evolutionary search explores task sequences and programs. It reports large gains on LIBERO-Pro, Robo- suite and BEHAVIOR-1K. ENPIRE builds a harness in which coding agents improve real-world robot policies [6]. Its mod- ules handle automatic environment reset and verification, pol- icy training, parallel physical rollouts, and an evolution step in which agents read logs and literature and revise training code. With it, agents reach near-perfect success on dexterous real-world tasks. Both results support our view that the agent is capable and that the harness decides what it can achieve. ENPIRE treats automatic reset and verification as a module of its own, and ASPIRE’s engine is built to make failures diagnos- able; these are the representability and resolvability conditions in our terms. Our study complements them with a long, fully recorded run that did not reach its goal: a fifteen-step house- hold task with open-vocabulary relational instructions, where perception semantics, skill chains and a human-maintained harness became the limiting factors. We report why, not a benchmark gain. Language-conditioned robot programs and grounding. Code as Policies [18], SayCan [19] and VoxPoser [20] com- pose language models with perception and control, typi- cally with hand-built perception. Open-vocabulary segmen- tation [8,9,16] and spatially grounded vision-language mod- els [10–12] provide the building blocks. Semantic displace- ment (§5) describes what happens when a code-writing im- prover has the former but not the latter: it rebuilds relational reasoning from geometry and does not converge. 9 Table 3: Design implications and how to falsify them. Every experiment should use frozen library snapshots, matched agent and simulator budgets, repeated seeds, and an evaluation set the loop has never selected against. ImplicationConditionFalsifying experiment D1 Attribute by capabilityRepr.Re-label the recorded archive with capability-based attribution; count gaps that move to a different family, and compare proposal success on relabelled vs. original gaps. D2 Measured embodimentRepr.Run the loop with declared vs. probed actuator descriptions; compare how often candidates hit evaluator boundaries (e.g. values piling at an advertised constant). D4 Reason, then verifyExpr. Replace only the region grounder in a frozen library; measure region success and chain depth on new seeds and new kitchen layouts. D5 Model-class escalationExpr.Repair-only proposer vs. proposer with an escalation policy at equal budget; rounds to the first fresh-state win on a relational front. D6 Chain in the substrateExpr.Fixed programs vs. gated chain edits; per-step reach and conditional reliability, and task success. D7 States past the frontierResol.Task-start test states vs. states reset to the changed step; fraction of evaluable pairs and detection rate of known-good changes. D8 Statistical gatesResol.Current vs. resolution-aware gates on the same candidate stream; rate of accepted changes that improve the lockbox. D9 Novelty budgetResol.Fixed seeds vs. fixed panel plus new states; distinct failure mechanisms observed per round. D10 Typed memoryRecursionReplay proposers on the archive with true vs. mislabelled request status; re-proposal rate of useful requests. D11 Integration obligationRecursionRegistration alone vs. registration plus obligation; call rate on eligible states and paired task change. D12 Harness in the loopRecursionGate proposed harness changes by archive replay; fraction of later human fixes the loop would have found. Evaluation under selection. Reward misspecification [13], few-run statistical fragility [15] and adaptive reuse of hold- outs [14] are well studied in isolation. In a self-improvement loop they compound: each accepted change is selected by the evaluator it may be exploiting, and becomes part of the environment for the next round. 10 Limitations This is one long run on one simulator, one kitchen layout and three tasks, with no independent replicates, matched-budget baselines or ablations. Harness repairs, model versions and providers changed along the same timeline, so our analysis identifies mechanisms from records and dissections, not causal effect sizes. We cannot say whether a different agent, more compute, or a different library representation would have solved the task. The vision-language check in §5 covers a handful of recorded views and shows feasibility, not bench- mark performance. The framework of §4 is deliberately simple. Its value is diagnostic: it tells which part of a loop to examine when accepted changes stop moving the task. It is not a predic- tive model. The design implications in §8 are hypotheses, and Table 3 states how each could be refuted. The appendix gives a round-by-round record of the run (Table 4) and the measure- ments behind each statement in the main text (Table 5). 11 Conclusion A coding agent given a robot, a simulator and a rule against human-written skills did discover real capabilities, and it was a sharper diagnostician of its own harness than we expected. It did not solve a household task, and the reasons were not a shortage of code or of reasoning. Relational meaning lived between perception modules, where only brittle code could reach it. Chains locked learning onto their first link. And the selection side of the loop, its evaluators, gates and memory, was a fallible model of the world that the agent exploited, and that only humans could improve. Self-improvement in software rests on representable failures, expressible fixes and resolvable tests. In robotics these must be built, and we argue that building them, not stronger proposers, is where progress will come from. Until then, the number of accepted changes measures activity, not progress. References [1]Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandku- mar. Voyager: An open-ended embodied agent with large lan- guage models. arXiv preprint arXiv:2305.16291, 2023. URL https://arxiv.org/abs/2305.16291. [2]Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models.arXiv preprint 10 arXiv:2310.12931, 2023. URLhttps://arxiv.org/abs/ 2310.12931. [3] Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, and Jeff Clune. Darwin Gödel Machine: Open-ended evolution of self- improving agents. arXiv preprint arXiv:2505.22954, 2025. URL https://arxiv.org/abs/2505.22954. Version 3, March 2026. [4] Karim Elmaaroufi, Justin Svegliato, Sarunas Kalade, Graham Schelle, Sanjit A. Seshia, and Matei Zaharia. RHO: Your coding agent is secretly a roboticist. arXiv preprint arXiv:2606.16458, 2026. URL https://arxiv.org/abs/2606.16458. [5]Runyu Lu, Yubo Wu, Ethan Kou, Letian Fu, Wenli Xiao, Ajay Mandlekar, Yinzhen Xu, Guanya Shi, Ken Goldberg, Ang Chen, Mosharaf Chowdhury, Yuke Zhu, Linxi Fan, and Guanzhi Wang. ASPIRE: Agentic /skills discovery for robotics. arXiv preprint arXiv:2607.00272, 2026. [6]Wenli Xiao, Jia Xie, Tonghe Zhang, Haotian Lin, Letian Fu, Haoru Xue, Jalen Lu, Yi Yang, Cunxi Dai, Zi Wang, Jimmy Wu, Guanzhi Wang, S. Shankar Sastry, Ken Goldberg, Linxi Fan, Yuke Zhu, and Guanya Shi. ENPIRE: Agentic robot policy self-improvement in the real world. arXiv preprint arXiv:2606.19980, 2026. [7] Soroush Nasiriany, Abhiram Maddukuri, Lance Zhang, Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Mandlekar, and Yuke Zhu. RoboCasa: Large-scale simulation of everyday tasks for generalist robots. In Robotics: Science and Systems, 2024. URL https://arxiv.org/abs/2406.02523. [8] Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, et al. SAM 3: Segment anything with concepts. arXiv preprint arXiv:2511.16719, 2025. [9]Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence- 2: Advancing a unified representation for a variety of vision tasks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. [10]Shuai Bai, Keqin Chen, Xuejing Liu, et al. Qwen2.5-VL techni- cal report. arXiv preprint arXiv:2502.13923, 2025. [11]Boyuan Chen, Zhuo Xu, Sean Kirmani, Brian Ichter, Danny Driess, Pete Florence, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. SpatialVLM: Endowing vision-language models with spatial reasoning capabilities. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. [12]Wentao Yuan, Jiafei Duan, Valts Blukis, Wilbert Pumacay, Ran- jay Krishna, Adithyavairavan Murali, Arsalan Mousavian, and Dieter Fox. RoboPoint: A vision-language model for spatial affordance prediction for robotics. In Conference on Robot Learning, 2024. [13]Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565, 2016. [14]Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Aaron Roth. Generalization in adaptive data analysis and holdout reuse. arXiv preprint arXiv:1506.02629, 2015. URL https://arxiv.org/abs/1506.02629. [15]Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C. Courville, and Marc G. Bellemare. Deep reinforce- ment learning at the edge of the statistical precipice. In Advances in Neural Information Processing Systems, volume 34, pages 29304–29320, 2021. URLhttps://arxiv.org/abs/2108. 13264. [16]Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. In IEEE/CVF International Con- ference on Computer Vision, 2023. [17] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, 2023. [18]Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Haus- man, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In IEEE International Conference on Robotics and Automation, 2023. URL https://arxiv.org/abs/2209.07753. [19]Michael Ahn, Anthony Brohan, Noah Brown, et al. Do as I can, not as I say: Grounding language in robotic affordances. In Conference on Robot Learning, 2022. [20] Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. VoxPoser: Composable 3D value maps for robotic manipulation with language models. In Conference on Robot Learning, 2023. URLhttps://arxiv.org/abs/2307. 05973. 11 A Round-by-round record Table 4 lists every round: how the tasks went, which problems the proposer worked on, what it proposed and how those proposals ended, which skill changes were accepted, which external models were registered, and which harness problems the round exposed. It shows the three phases of §3 in detail. Early rounds make and accept changes on the fridge task. The middle rounds turn up harness problems one after another. The late rounds keep working on the same four problems (find object, find region, grasp, reach) while registering new models, without changing the outcome. Table 4: Round-by-round record. Outcome: task successes / completed rollouts for FRIDGE (F), CABINET (C) and SINK (S). Problems worked on: the gaps sent to the proposer, as step: diagnosis. Proposals: attempts, of which research requests (req.), declines (no change) and rejections by test (replay = exact failure replay; fresh = paired fresh-scene and held-out test; transfer = sealed recorded failures; regr/inv = regression and invariance checks). Accepted: skill changes that passed every test, with the problem they fixed. Models: external models registered by the research agent, and research jobs that did not register. Harness problems found: harness repairs made by humans after this round exposed the problem. Rd OutcomeProblems worked onProposalsAccepted skill changesModelsHarness problems found 1 F 0/4find obj: wrong object1 att.; rej. replay 1– 2 F 0/4find obj: no task object near point 2 att.; rej. replay 2–target_not_in_view attributed on visibility 3 F 0/4find obj: no task object near point 2 att.; rej. replay 2– effect packs visible through worker wrappers 4 F 0/4find obj: target not in view2 att.; 1 req.; rej. replay 1 –1 not reg.agent-extensible provider kinds 5 F 0/4find obj: target not in view2 att.; 1 req.; rej. replay 1 –target_free_active_view_ base_placement – 6 F 0/4find region: wrong fixture2 att.; rej. replay 1, fresh 1 – 7 F 0/4find obj: target not in view; find region: wrong fixture 4 att.; rej. replay 4– full diffs kept in candidate memory 8 F 0/4find obj: target not in view; find region: wrong fixture 4 att.; rej. replay 4–counterfactual gate semantics documented 9 F 0/4find obj: target not in view; find region: wrong fixture 4 att.; rej. replay 4–trial execution traces fed back 10 F 0/4find obj: target not in view; find region: wrong fixture 3 att.; rej. replay 2fixture_region routing, fixture_ region.deployed_localize (find region: wrong fixture) – 11 F 0/4find obj: target not in view; find region: wrong fixture 3 att.; 1 no change; rej. replay 2 – 12 F 0/4find obj: target not in view; find region: wrong fixture 4 att.; 1 no change; rej. fresh 2, replay 1 – 13 F 0/4find obj: target not in view; find region: fixture not matched 6 att.; rej. replay 6–elitism seeding 14 F 0/4find obj: target not in view; find region: fixture not matched 6 att.; 1 req.; rej. replay 3, fresh 2 –equivariant_grid_base_ placement per-case fresh-state feedback 15 F 0/4find obj: target not in view; find region: fixture not matched 5 att.; 1 req.; rej. fresh 2, replay 1 localize.calibrated_multiview_rgb (find obj: target not in view) 1 not reg.– 16 F 0/4find obj: target not in view; find region: fixture not matched 4 att.; 1 no change; rej. replay 3 – 17 F 0/4find obj: target not in view; find region: fixture not matched; move: target not reached 9 att.; 1 req.; rej. replay 4, error 3 localize.calibrated_multiview_rgb (find obj: target not in view) 1 not reg.era-aware selective memory 18 F 0/4find obj: no task object near point; find region: fixture not matched; move: target not reached 7 att.; 1 req.; 1 no change; rej. replay 5 –target_graspgen_6dof– 19 F 0/4find obj: target not in view; find region: fixture not matched; move: target not reached 7 att.; 1 req.; rej. replay 5 localize.calibrated_multiview_rgb (find obj: target not in view) relational_rgbd_localizer– 20 F 0/4find obj: target not found; find region: wrong fixture; move: target not reached 5 att.; rej. replay 3fixture_region.deployed_localize (find region: wrong fixture); localize. sam3_1_da3 (find obj: target not found) – 21 F 0/4find obj: target not found; find region: fixture not matched; move: target not reached 6 att.; 1 req.; 1 no change; rej. replay 4 –mplib_whole_body_goalset– 22 F 0/4find obj: target not found; find region: fixture not matched; move: target not reached 6 att.; 1 req.; 1 no change; rej. replay 4 –relational_rgbd_localizer– 12 Rd OutcomeProblems worked onProposalsAccepted skill changesModelsHarness problems found 23 F 0/4find obj: target not in view; find region: fixture not matched; move: target not reached 7 att.; 1 req.; rej. replay 5, fresh 1 –mplib_confined_motion– 24 F 0/4find obj: program exception; find region: fixture not matched; move: target not reached 9 att.; rej. replay 9–fixture-region evaluator truthful (refrigerator/fridge, 3-D containment) 25 F 0/4– 26 F 0/4find obj: no task object near point; find region: wrong fixture; move: target not reached 6 att.; 1 req.; rej. replay 3, fresh 1 localize.calibrated_multiview_rgb (find obj: no task object near point) equivariant_grid_base_ placement – 27 F 0/4find obj: no task object near point; find region: wrong region; move: target not reached 6 att.; 1 req.; 1 no change; rej. replay 4 –equivariant_grid_base_ placement evaluator change bounds prior-attempt era 28 F 0/4find obj: no task object near point; find region: wrong region; move: target not reached 4 att.; 2 req.; 1 no change; rej. replay 1 –2 not reg.materialization failures enter refinement; probe synthesizes proprio/control inputs 29 F 0/4find obj: no task object near point; find region: wrong region; move: target not reached 7 att.; 1 req.; 1 no change; rej. replay 3, fresh 2 –relational_rgbd_localizer– 30 F 0/4find obj: no task object near point; find region: wrong region; move: target not reached 4 att.; 1 req.; 1 no change; rej. replay 1 localize routing, localize.calibrated_ multiview_rgb (find obj: no task object near point) relational_rgbd_localizerprovider repairs do not fork providers 31 F 0/4find obj: program exception; find region: wrong region; move: target not reached 7 att.; 1 no change; rej. replay 6 – 32 F 0/4find obj: program exception; find region: wrong region; move: target not reached 7 att.; 1 no change; rej. replay 5, fresh 1 – 33 F 0/4find obj: program exception; find region: wrong region; move: target not reached 7 att.; 1 no change; rej. replay 6 –baseline replay control; unreplayable segments retire 34 F 0/4find obj: program exception; find region: wrong region; move: target not reached 6 att.; 1 req.; 2 no change; rej. replay 3 –relational_rgbd_localizer– 35 F 0/4find obj: program exception; find region: wrong region; move: target not reached 5 att.; 2 no change; rej. replay 3 – 36 F 0/4find obj: program exception; find region: wrong region; move: target not reached 3 att.; 2 no changefixture_region.deployed_localize (find region: wrong region) – 37 F 0/4find obj: target not in view; find region: wrong region; move: target not reached 7 att.; 1 no change; rej. replay 6 –controller-grounded reachable evaluator; real actuator surface (torso, layout, IK); upstream evaluator changes expire evidence 38 F 0/4find obj: target not in view; find region: wrong region; move: target not reached 5 att.; 1 req.; 1 no change; rej. fresh 3 –relational_rgbd_localizer– 39 F 0/4find obj: target not in view; find region: wrong region 9 att.; rej. replay 5, fresh 3 localize.calibrated_multiview_rgb (find obj: target not in view) – 40 F 0/4find obj: target not in view; find region: wrong region 3 att.; 1 no changefixture_region.deployed_localize (find region: wrong region); localize. calibrated_multiview_rgb (find obj: target not in view) – 41 F 0/4find obj: target not in view; find obj: wrong object; find region: wrong region 3 att.; 1 no changelocalize.calibrated_multiview_rgb (find obj: target not in view); localize.calibrated_multiview_rgb (find obj: wrong object) – 42 F 0/4find obj: no task object near point; find obj: wrong object; find region: wrong region 3 att.; 1 no changelocalize.sam3_1_da3 (find obj: no task object near point); localize routing, localize.calibrated_ multiview_rgb (find obj: wrong object) – 43 F 0/4find region: wrong fixture; find region: wrong region; reach: target out of workspace 7 att.; 1 req.; 1 no change; rej. fresh 2, replay 2 fixture_region routing, fixture_ region.deployed_localize (find region: wrong region) mplib_free_space_ reachability – 44 F 0/4find obj: no task object near point; find obj: target not in view; find region: wrong region 5 att.; 1 no change; rej. replay 3 localize routing, localize.calibrated_ multiview_rgb (find obj: no task object near point) –episode budget sized to task horizon; aborted subgoals keep evaluator residual 45 F 0/4find obj: no task object near point; find obj: target not in view; find region: wrong region 4 att.; 2 no change; rej. replay 1 localize.calibrated_multiview_rgb (find obj: no task object near point) – 13 Rd OutcomeProblems worked onProposalsAccepted skill changesModelsHarness problems found 46 F 0/4find region: wrong region; reach: target out of workspace 5 att.; 1 req.; rej. replay 2 fixture_region.deployed_localize (find region: wrong region); fixture_ region routing, fixture_region. deployed_localize (find region: wrong region) mplib_free_space_ reachability step cost is evidence 47 F 0/4find obj: no task object near point; find region: wrong region; reach: program exception 5 att.; 1 req.; rej. error 1, replay 1 fixture_region.deployed_localize (find region: wrong region); localize. calibrated_multiview_rgb (find obj: no task object near point) mplib_free_space_ reachability codex executable fallback 48 F 0/4find obj: no task object near point; find obj: target not in view; find region: fixture not found 8 att.; rej. replay 7localize.sam3_1_da3 (find obj: no task object near point) –program wall clock scales with step budget; unattributed task failure triggers revision 49 F 0/4find obj: target not in view; find region: fixture not found; find region: wrong region 9 att.; rej. replay 9–target geometry attached to not-in-view verdicts 50 F 0/4find obj: target not in view; find region: fixture not found; find region: wrong region 7 att.; rej. error 4, replay 2 localize routing, localize.calibrated_ multiview_rgb (find obj: target not in view) – 51 F 0/4find obj: target not in view; find obj: wrong object 9 att.; rej. error 9– 52 F 0/4find obj: target not in view; reach: target out of workspace 6 att.; 1 req.; rej. replay 4 localize.calibrated_multiview_rgb (find obj: target not in view) mplib_free_space_ reachability – 53 F 0/4find obj: no task object near point; find obj: target not found; find region: wrong region 5 att.; rej. replay 3; infra/unrepl. 1 fixture_region.deployed_localize (find region: wrong region) –mining bounded to recent library generations 54 F 0/4find region: fixture not found; find region: wrong region; reach: target out of workspace 4 att.; 1 req.; 1 no change; rej. replay 1; infra/unrepl. 1 –mplib_free_space_ reachability raising calls are branch targets 55 F 0/4find obj: target not in view; find obj: wrong object 5 att.; rej. replay 2localize.calibrated_multiview_rgb (find obj: target not in view); localize.calibrated_multiview_rgb (find obj: wrong object); localize. calibrated_multiview_rgb (find obj: wrong object) –structured diagnostics reach proposer 56 F 0/5find obj: no task object near point; find obj: target not in view; reach: target out of workspace 6 att.; 1 req.; rej. replay 4 localize.calibrated_multiview_rgb (find obj: no task object near point) mplib_free_space_ reachability base servo dead zone; truthful speed model 57 F 0/4find obj: target not in view; find region: no fixture near point; reach: target out of workspace 7 att.; rej. replay 6fixture_region.deployed_localize (find region: no fixture near point) –torso servo; truthful lift model 58 F 0/4find obj: target not found; find region: fixture not found; reach: target out of workspace 7 att.; rej. replay 6localize routing, localize.calibrated_ multiview_rgb (find obj: target not found) –base stall watchdog; contact kind reported 59 F 0/4find obj: wrong object; grasp: no grasp contact; reach: target out of workspace 9 att.; rej. replay 7, fresh 2 –native action layout advertises measured response; concurrent gap pipelines; effect gate cannot fail an unexercised family 60 F 0/4 C 0/4find obj: target not in view; find region: fixture not found; grasp: no grasp contact; reach: target out of workspace 10 att.; rej. replay 5, fresh 4 fixture_region.deployed_localize (find region: fixture not found) – 61 F 0/4 C 0/4find obj: wrong object; find region: fixture not found; grasp: closed above object; reach: target out of workspace 12 att.; rej. replay 9, fresh 3 –elitism seeds from best partial rescue; strict paired improvement waives absolute floor 62 F 0/4 C 0/4find obj: wrong object; find region: fixture not found; grasp: closed above object; reach: target out of workspace 12 att.; rej. fresh 7, replay 4 approach.metric_reach_plan (reach: target out of workspace) – 63 F 0/5 C 0/4find obj: wrong object; find region: fixture not found; grasp: closed above object 10 att.; rej. replay 6, fresh 2, regr 1 grasp routing, grasp.analytic_top (grasp: closed above object) –orchestrator program memory 64 F 0/4 C 0/4find obj: no task object near point; find region: no fixture near point; reach: target out of workspace 10 att.; rej. replay 5, fresh 1, stale 1, regr 1 fixture_region.deployed_localize, fixture_region.metric_masks (find region: no fixture near point); approach.metric_reach_plan (reach: target out of workspace) – 65 F 0/4 C 0/4find obj: no task object near point; find region: no fixture near point; find region: wrong region; reach: target out of workspace 10 att.; 1 req.; rej. replay 3, stale 1, error 1, fresh 1 fixture_region.deployed_localize (find region: no fixture near point); fixture_region.deployed_localize (find region: wrong region); localize. calibrated_multiview_rgb (find obj: no task object near point) mplib_free_space_ reachability provider process recovery; unmeasured pairs excluded; fallback labels; gate baselines follow live library; three support tasks; sealed holdout task; program memory keyed by archived source; honest routing-fit partitions; reach-trial budget 14 Rd OutcomeProblems worked onProposalsAccepted skill changesModelsHarness problems found 66 F 0/1 C 0/3 S 0/2 find obj: wrong object; find region: no fixture near point; grasp: no grasp contact; reach: target out of workspace 9 att.; rej. replay 3, fresh 2; infra/unrepl. 3 fixture_region routing, fixture_ region.deployed_localize (find region: no fixture near point) mplib_free_space_ reachability provider health shown to proposer; harness-filed provider repairs 67 F 0/4 C 0/4 S 0/4 find obj: target not in view; find obj: wrong object; grasp: no grasp contact; reach: target out of workspace 8 att.; 1 req.; rej. replay 4, fresh 1 localize.calibrated_multiview_rgb (find obj: target not in view); approach.hold (reach: target out of workspace); routing refit relational_rgbd_localizer– 68 F 0/4 C 0/4 S 0/4 find obj: target not in view; find region: wrong region; grasp: closed beside object; reach: target out of workspace 11 att.; 1 req.; rej. replay 8, fresh 2 routing refittarget_graspgen_6dofnumpy scalar hashing; fake simulator exposes torso 69 F 0/4 C 0/4 S 0/4 find obj: target not in view; find region: wrong region; grasp: closed beside object; reach: target out of workspace 12 att.; rej. replay 8, fresh 4 routing refit–generalization gap needs enough holdout cases 70 F 0/4 C 0/4 S 0/4 find obj: target not in view; find region: fixture not found; grasp: closed above object; reach: target out of workspace 12 att.; 1 req.; rej. replay 9, fresh 2 routing refitmplib_confined_motion– 71 F 0/4 C 0/4 S 0/4 find obj: target not in view; find region: fixture not found; grasp: closed beside object; reach: target out of workspace 9 att.; rej. replay 6, fresh 1 fixture_region.deployed_localize (find region: fixture not found); localize.calibrated_multiview_rgb (find obj: target not in view); routing refit – 72 F 0/4 C 0/4 S 0/4 find obj: target not in view; find region: no fixture near point; grasp: closed above object; reach: target out of workspace 12 att.; rej. replay 10, fresh 1 fixture_region.deployed_localize (find region: no fixture near point); routing refit – 73 F 0/4 C 0/4 S 0/4 find obj: no task object near point; find region: no fixture near point; grasp: closed above object; reach: target out of workspace 9 att.; 1 req.; rej. replay 5 fixture_region.deployed_localize (find region: no fixture near point); localize.calibrated_multiview_rgb (find obj: no task object near point); approach.hold, approach.metric_ reach_plan (grasp: closed above object); routing refit mplib_free_space_ reachability view precondition judged at block start; recorded rescues outrank precondition ownership; reach failures split by mechanism; empty provider answers count as starvation; agent container isolation; transfer gate and unseen-win rule; sensor.depth not privileged 74 F 0/4 C 0/4 S 0/4 find obj: wrong object; find region: wrong fixture; move: target not reached; reach: tool path blocked 10 att.; 1 req.; rej. replay 5, error 1, transfer 1, fresh 1; infra/unrepl. 1 routing refitmplib_whole_body_goalset, relational_rgbd_localizer; 1 not reg. – 75 F 0/4 C 0/4 S 0/4 find obj: target not in view; find region: wrong fixture; move: target not reached; reach: tool path blocked 11 att.; 1 req.; rej. replay 5, fresh 3, transfer 1 fixture_region.deployed_localize (find region: wrong fixture); routing refit equivariant_grid_base_ placement, relational_rgbd_ localizer; 1 not reg. generalization gap compares like residuals; second fresh draw; side-effects gate; repairs replayed on recorded inputs; self-filtered depth 76 F 0/4 C 0/4 S 0/4 find obj: wrong object; find region: wrong region; move: target not reached; reach: program exception 7 att.; 2 no change; rej. replay 4 fixture_region.deployed_localize (find region: wrong region); routing refit mplib_free_space_ reachability, mplib_whole_ body_goalset program faults not mining evidence; provider time excluded from program wall budget 77 F 0/4 C 0/4 S 0/4 find obj: wrong object; find region: wrong region; move: target not reached; reach: tool path blocked 7 att.; 3 no change; rej. replay 4 routing refitloftr_multiview_localization, mplib_whole_body_goalset tool-mount probe retried; motion failures split precondition vs motion; masked implementation names requirement; replay gate sample size 78 F 0/4 C 0/4 S 2/4 find obj: target not in view; find region: wrong region; move: target not reached; reach: tool path blocked 8 att.; 2 no change; rej. replay 6 routing refitmplib_whole_body_goalset; 1 not reg. – 79 F 0/4 C 0/4 S 2/4 find obj: wrong object; find region: wrong region; move: target not reached; reach: tool path blocked 8 att.; 2 no change; rej. replay 6 routing refit1 not reg.rotate away from declined kinds; routing credibility scales with gap size 80 F 0/4 C 0/4 S 1/4 find obj: wrong object; find region: wrong region; grasp: localization offset; move: tool path blocked 8 att.; 2 no change; rej. replay 6 routing refit1 not reg.– 81 F 0/4 C 0/4 S 1/4 find obj: target not in view; find region: wrong region; grasp: closed beside object; reach: tool path blocked 8 att.; 2 no change; rej. replay 4, regr 1, fresh 1 routing refitloftr_multiview_localization; 1 not reg. keep successful programs 82 F 0/4 C 0/4 S 2/4 find obj: target not in view; find region: no fixture near point; find region: wrong region; move: tool path blocked 10 att.; 1 no change; rej. replay 9 routing refit2 not reg.– 83 F 0/4 C 0/4 S 1/4 find obj: wrong object; find region: wrong region; move: target not reached; reach: tool path blocked 8 att.; 2 no change; rej. replay 5, fresh 1 routing refit1 not reg.– 15 Rd OutcomeProblems worked onProposalsAccepted skill changesModelsHarness problems found 84 F 0/4 C 0/4 S 0/4 find obj: target not in view; find region: no fixture near point; find region: wrong region; grasp: localization offset 10 att.; 1 no change; rej. replay 9 routing refitloftr_multiview_localizationrouting changes effect-gated; repair cooldown 85 F 0/4 C 0/4 S 0/4 find obj: target not in view; find region: wrong region; move: target not reached; reach: tool path blocked 8 att.; 2 no change; rej. replay 6 –mplib_free_space_ reachability – 86 F 0/4 C 0/4 S 0/4 find obj: target not in view; find region: no fixture near point; find region: wrong region; move: tool path blocked 10 att.; 1 no change; rej. replay 9 routing refit–stalled kinds rotate out 87 F 0/4 C 0/4 S 0/4 find obj: target not in view; find region: wrong fixture; grasp: closed beside object; reach: tool path blocked 8 att.; 2 no change; rej. replay 6 routing refitloftr_multiview_localization– 88 F 0/4 C 0/4 S 0/4 find obj: target not in view; find region: no fixture near point; find region: wrong region; move: tool path blocked 10 att.; 1 no change; rej. replay 8, fresh 1 routing refit1 not reg.replay early stop; rejected repairs cool down 89 F 0/4 C 0/4 S 0/4 find obj: wrong object; find region: wrong fixture; find region: wrong region; reach: tool path blocked 8 att.; 1 no change; rej. replay 6 localize routing, localize.calibrated_ multiview_rgb (find obj: wrong object); routing refit mplib_whole_body_goalset; 1 not reg. provider registration effect-gated 90 F 0/4 C 0/4 S 0/4 find obj: target not in view; find region: wrong region; grasp: no grasp contact; move: target not reached 7 att.; 3 no change; rej. replay 4 –loftr_multiview_localizationcomposing families effect-gated 91 F 0/4 C 0/4 S 0/4 find obj: wrong object; find region: wrong fixture; grasp: localization offset; reach: tool path blocked 6 att.; 3 no change; rej. replay 3 routing refit– 92 F 0/4 C 0/4 S 0/4 find obj: target not in view; find region: wrong fixture; find region: wrong region; grasp: closed above object 8 att.; 2 no change; rej. replay 6 routing refit–region visibility judged at block start 93 F 0/4 C 0/4 S 1/4 find obj: wrong object; grasp: localization offset; reach: tool path blocked 4 att.; 4 no changerouting refit1 not reg.– 94 F 0/4 C 0/4 S 0/4 find obj: target not in view; find region: wrong region; grasp: no grasp contact; move: target not reached 5 att.; 4 no change; rej. replay 1 routing refit–proposer prompted for evidence-backed attempts 95 F 0/4 C 0/4 S 0/4 find region: no fixture near point; find region: wrong fixture; grasp: localization offset; reach: tool path blocked 8 att.; 4 no change; rej. replay 4 routing refit– 96 F 0/4 C 0/4 S 0/4 find obj: target not in view; find obj: wrong object; find region: wrong region; grasp: no grasp contact 12 att.; 1 no change; rej. replay 9, inv 1 localize.calibrated_multiview_rgb (find obj: wrong object); routing refit 2 not reg.– 97 F 0/4 C 0/4 S 0/4 find obj: target not in view; find region: wrong fixture; grasp: no grasp contact; reach: tool path blocked 10 att.; 2 no change; rej. replay 6, regr 1, fresh 1 routing refit1 not reg.– 98 F 0/4 C 0/4 S 1/4 find obj: target not in view; find region: wrong region; grasp: localization offset; grasp: no grasp contact 7 att.; 2 no change; rej. replay 2, regr 1, inv 1; infra/unrepl. 1 routing refit–branch numbering across nested calls; interpreter mounts in container 99 F 0/4 C 0/4 S 1/4 find obj: target not found; find region: fixture not found; find region: wrong fixture; reach: tool path blocked 8 att.; 3 no change; rej. fresh 3, replay 2 routing refit2 not reg.CUDA probe timeout; observer export detached 100 C 0/4 S 0/4find obj: no task object near point; find obj: target not in view; grasp: closed above object; grasp: localization offset 6 att.; 2 no change; rej. replay 3 localize.sam3_1_da3 (find obj: target not in view); routing refit 2 not reg. nested-family branching; diffs skip tooling trees 101 F 0/4 C 0/4 S 1/4 find region: wrong region; grasp: no grasp contact; move: tool path blocked; reach: tool path blocked 6 att.; 3 no change; rej. replay 3 routing refit1 not reg.declared/commanded target mismatch is program fault 16 Rd OutcomeProblems worked onProposalsAccepted skill changesModelsHarness problems found 102 F 0/4 C 0/4 S 0/4 find obj: no task object near point; find region: wrong fixture; grasp: no grasp contact 8 att.; 2 no change; rej. replay 6 routing refit1 not reg.target mismatch triggers orchestrator revision; query naming a fixture is a program fault; partial rescues reach fresh-state gates; stall rotation follows fresh-state gates; proximal faults classify small gaps; orchestrator sees observed usage 103 F 0/4 C 0/5 S 0/4 find region: wrong region; grasp: no grasp contact; move: tool path blocked; reach: tool path blocked 10 att.; 1 no change; rej. replay 9 routing refit–no validation program: not applicable; requests under phi/requests recorded 104 F 0/4 C 0/4 S 0/4 find region: wrong fixture; grasp: localization offset; move: tool path blocked; reach: tool path blocked 9 att.; 2 no change; rej. replay 7 routing refitloftr_multiview_localization; 2 not reg. identified mechanism ranks first; paired differences must reproduce 105 F 0/4 C 0/4 S 0/4 find region: wrong region; grasp: no grasp contact; move: tool path blocked; reach: tool path blocked 7 att.; 2 req.; 2 no change; rej. replay 3 routing refit2 not reg. measured rescue survives upstream evaluator change 106 F 0/4 C 0/4 S 0/4 find obj: target not in view; find region: wrong fixture; grasp: no grasp contact; reach: tool path blocked 8 att.; 1 req.; 1 no change; rej. regr 3, replay 2, fresh 1 routing refitequivariant_grid_base_ placement, relational_rgbd_ localizer; 1 not reg. probe suites import harness 107 F 0/4 C 0/4 S 1/4 find obj: target not in view; find region: wrong region; move: tool path blocked; reach: tool path blocked 7 att.; 2 req.; 1 no change; rej. replay 2, fresh 1 localize.calibrated_multiview_rgb (find obj: target not in view); routing refit observed_serial_pose_ transport; 2 not reg. draw that cannot show repair does not reject 108 F 0/4 C 0/4 S 0/4 find obj: target not in view; grasp: no grasp contact; move: tool path blocked; reach: tool path blocked 8 att.; 2 req.; rej. fresh 4, replay 2 routing refitmplib_whole_body_goalset; 2 not reg. evaluator refs cached; task-span rule over exercised tasks 109 F 0/4 C 0/4 S 0/4 find obj: target not in view; find region: wrong region; move: tool path blocked; reach: tool path blocked 6 att.; 2 req.; 1 no change; rej. replay 3 routing refitobserved_serial_pose_ transport; 1 not reg. – 110 F 0/4 C 0/4 S 0/4 find obj: target not in view; grasp: no grasp contact; move: tool path blocked; reach: tool path blocked 8 att.; 2 req.; rej. replay 3, fresh 2, transfer 1 routing refitloftr_multiview_localization, mplib_confined_motion, observed_serial_pose_ transport, relational_rgbd_ localizer; 1 not reg. real wins count at transfer; requests never lost; truthful ledger; decline backoff; repair replayed for what was asked; provider hosts over ssh; grounding_proposals capability class; memory budget above interpreter baseline; single-threaded program child 111 F 0/3 C 0/4 S 0/4 find region: wrong fixture; grasp: no grasp contact; move: tool path blocked; reach: tool path blocked 6 att.; 3 req.; rej. fresh 3 routing refitmplib_whole_body_goalset; 7 not reg. requests not judged by harness faults 112 F 0/4 C 0/4 S 0/4 find region: wrong fixture; grasp: no grasp contact; move: tool path blocked; reach: tool path blocked 6 att.; 3 req.; rej. fresh 2, transfer 1 routing refitequivariant_grid_base_ placement, observed_serial_ pose_transport; 4 not reg. residual-matched sealed transfer states 113 F 0/4 C 0/4 S 0/4 find obj: target not in view; grasp: no grasp contact; move: tool path blocked; reach: tool path blocked 8 att.; 2 req.; rej. replay 3, fresh 3 routing refitloftr_multiview_localization, mplib_confined_motion, mplib_free_space_ reachability; 4 not reg. – 114 F 0/4 C 0/4 S 0/4 find obj: target not in view; find region: wrong region; move: tool path blocked; reach: tool path blocked 6 att.; 3 req.; rej. replay 2, fresh 1 routing refitimage_expression_ semantic_assessor, visual_ referent_entailment; 4 not reg. – 115 F 0/4 C 0/4 S 0/4 find obj: target not in view; find region: wrong region; move: tool path blocked; reach: tool path blocked 6 att.; 3 req.; rej. replay 3 routing refitmplib_confined_motion, observed_transport_ assessment, visual_referent_ entailment; 3 not reg. – 116 F 0/4 C 0/4 S 0/4 find obj: target not in view; find region: wrong region; move: tool path blocked; reach: tool path blocked 6 att.; 3 req.; rej. transfer 1, replay 1, fresh 1 routing refitdiscriminative_region_ association, discriminative_ region_compatibility, mplib_ confined_motion, relational_ rgbd_localizer; 5 not reg. – 117 F 0/4 C 0/4 S 0/4 find region: wrong region; grasp: no grasp contact; move: tool path blocked; reach: tool path blocked 6 att.; 2 req.; 1 no change; rej. fresh 3 routing refitmplib_confined_motion, observed_transport_ assessment, relational_rgbd_ localizer – 118 F 0/4 C 0/4 S 0/4 find region: wrong region; grasp: no grasp contact; move: tool path blocked; reach: tool path blocked 4 att.; 3 req.grasp.analytic_top, grasp.rgbd_6dof_ retain (grasp: no grasp contact); routing refit mplib_confined_motion, observed_region_relational_ localizer, observed_ transport_assessment, relational_rgbd_localizer; 1 not reg. – 17 Rd OutcomeProblems worked onProposalsAccepted skill changesModelsHarness problems found 119 F 0/4 C 0/4 S 0/4 find obj: target not in view; find region: wrong region; grasp: closed beside object; reach: target out of workspace 10 att.; 1 req.; rej. replay 9 routing refitgrounded_clause_program, observed_region_relational_ localizer, relational_rgbd_ localizer, structured_ referent_hypotheses; 3 not reg. – 120 F 0/4 C 0/4 S 0/4 find obj: target not in view; find region: wrong region; grasp: closed beside object; reach: tool path blocked 8 att.; 3 req.; rej. replay 5 routing refitlanguage_spatial_search_ memory, mplib_confined_ motion, relational_rgbd_ localizer; 4 not reg. – 121 F 0/4 C 0/4 S 0/4 find obj: target not in view; find region: wrong region; grasp: no grasp contact; reach: target out of workspace 8 att.; 2 req.; rej. transfer 3, replay 2, fresh 1 routing refitdiscriminative_observed_ pairs, selective_context_ region_assessor; 3 not reg. – 122 F 0/4 C 0/4 S 0/4 find obj: target not in view; find region: wrong region; grasp: closed above object; reach: target out of workspace 10 att.; 1 req.; rej. replay 5, transfer 2, fresh 2 routing refittarget_graspgen_6dof; 2 not reg. – 123 F 0/4 C 0/4 S 0/4 find obj: target not in view; find region: wrong region; grasp: no grasp contact; reach: tool path blocked 10 att.; 1 req.; rej. replay 8, fresh 1 routing refit– B Measurements behind the main text The main text describes mechanisms; Table 5 gives the measurements behind each statement. Table 6 summarizes how the set of tasks and the harness changed over the run. The first accepted change of the whole run was a new skill family for finding regions (round 10); the other accepted changes repaired or extended existing skills. Table 5: Measurements behind the main-text claims. Rollout counts use completed rollouts; “late” means rounds 103–123; “three-task phase” means rounds 66–123. ClaimMeasurement Target task never completedFRIDGE 0/486; CABINET 0/256; SINK 14/230, of which 9 had failed intermediate steps Activity57 accepted library changes (51 perception); 86 provider registrations over 20 identities; 847 proposal attempts Region grounding is the dominant first failureFirst failed effect in 132/224 FRIDGE and 113/232 CABINET rollouts (three-task phase); wrong_region in all 62 late rollouts of fridge seeds 68, 70, 71 Semantic displacementRegion skill: 3,056 lines, 33 thresholds; composed localizer: 5,032 lines, 48 thresholds; 27 attempts on the two-state fridge gap rescued seed 69 in every trial and seed 68 in none Visibility entanglementTop-shelf occupant pixels at the region block’s start: about 3,000 / 1,273 / 292 / 45 on seeds 68 / 71 / 70 / 69 VLM feasibilityQwen2.5-VL 7B placed “the top shelf” and “the shelf below” correctly in 4/4 recorded head views, about 1 s per query Frontier lockFRIDGE: 57% fail step 1; 2.7% pass four steps; 0.9% pass five. In round 60, 5 of 6 fresh grasp cases never reached the grasp Attenuated local winRound-118 grasp change: reproduced wins on a fresh support state and a held-out state; next 60 rollouts: 0 task successes Evaluator capture276reachablesuccesses at 0.849–0.850 m under a 0.85 m sphere; torso absent from the action description (34 proposer complaints); base speed advertised 0.0016 m/step, measured 0.037 Unresolvable selection119 rejected candidates had rescued at least one exact state; 22 promotions (rounds 40–65) with 0 wins and 0 losses; 16 of 36 re-run disagreeing pairs did not reproduce Evidence fixed pointIdentical fridge and cabinet per-seed outcomes in rounds 90–102 Memory corruption88 request-bearing proposals gated as code (46 on the fridge-region front); gap declined 15 times citing the lost request Capability without use11 provider identities first registered in rounds 114–121 have zero later collection calls Framing sensitivity 0/8 edits in rounds 93–94 vs. 11/12 in round 96 after one paragraph of proposer text changed Human effort113 numbered harness fixes (Fig. 5b) 18 Table 6: Phases of the run. Boundaries summarize changes in which tasks were run and in the harness; they are not randomized treatment groups. “Accepted” counts skill changes that passed every test; routing refits are not included. RoundsWhat changedRolloutsCompletedTask successesAccepted changes 1–59Fridge task only; discovery243237033 60–65Cabinet task added494908 66–102Sink task added; harness repairs4484341314 103–110Tests and memory repaired979711 111–118Lost requests recovered; move to new machine969501 119–123After the round-118 grasp change606000 Total9939721457 C External models and their use Table 7 lists every external model the research agent registered. Models registered early (active viewing, base placement, grasp generation, motion planning, relational localization) were called by deployed skills thousands of times. Every model first registered from round 107 on has no call in any later task rollout. This includes all the vision-language grounders built for “the top shelf” (§5). Table 7: Every external model the research agent registered, and whether the robot used it. Calls are counted in task rollouts of rounds after the model’s first registration; test and research calls are not counted. Repairs of a model are folded into one row. The top block was used by deployed skills; the bottom block, mostly the late vision-language grounders for “the top shelf”, was never called. ModelFirst registeredLater rolloutsCallsRollouts with calls target_free_active_view_base_placement5973520295 equivariant_grid_base_placement14935185110 target_graspgen_6dof18919316316 relational_rgbd_localizer199151863867 mplib_whole_body_goalset21907189189 mplib_confined_motion238995145 mplib_free_space_reachability43816586491 loftr_multiview_localization77553215212 observed_serial_pose_transport10719200 visual_referent_entailment11410800 image_expression_semantic_assessor11410800 observed_transport_assessment1159600 discriminative_region_compatibility1168400 discriminative_region_association1168400 observed_region_relational_localizer1186000 structured_referent_hypotheses1194800 grounded_clause_program1194800 language_spatial_search_memory1203600 discriminative_observed_pairs1212400 selective_context_region_assessor1212400 D Where the run ended Table 8 shows where each rollout of the final round stopped. Three fridge scenes fail at finding the top shelf before passing a single step; the fourth passes three steps and fails at the grasp. The cabinet task fails at finding its destination region, at grasping or at reaching. The sink task fails at seeing its target or at grasping. E Additional figures Figure 6 shows where proposed changes were rejected and how often paired differences between old and new skills reproduced when re-run. Most rejections come from replaying the exact recorded failure. Of the differences that were re-run, nearly half did not reproduce. Figure 7 shows recorded head-camera views from the fridge and sink tasks. 19 Table 8: Where each rollout stopped in the final round (123). Steps passed counts consecutive declared steps that succeeded before the first failure. No rollout succeeded. TaskScene seedSteps passedFirst failed stepDiagnosis FRIDGE680 fixture_region_localized wrong_region 693 retained no_grasp_contact 700 fixture_region_localized wrong_region 710 fixture_region_localized wrong_region CABINET683 retained no_grasp_contact 691 fixture_region_localized wrong_fixture 704 reachable tool_path_blocked 711 fixture_region_localized wrong_fixture SINK680 localized target_not_in_view 692 retained closed_above_object 702 retained no_grasp_contact 712 retained closed_above_object 0100200300400500600700800 Evaluations (labels: reached / failed) Static checks Invariance Regression Replay of failure Fresh scenes + held-out Sealed failures Other skills 657 / 0 657 / 2 655 / 8 647 / 490 157 / 90 17 / 10 6 / 0 (a) Where proposed changes were rejected Reached Failed ReproducedNot reproduced Unmeasured 0 5 10 15 20 Re-run disagreeing pairs 19 16 1 (b) Re-running a difference Figure 6: Where candidates were rejected, and how stable paired comparisons were. (a) How many proposed changes reached and failed each test; replaying the recorded failure rejects most of them. (b) Of 36 fresh-scene pairs where the old and new skill disagreed and were re-run, 19 reproduced, 16 did not, and one was unmeasured. (a) Fridge, seed 68, round 115(b) Fridge, seed 69, round 115 (c) Sink, seed 71, round 107 Figure 7: Recorded head-camera frames. (a) Start of the top-shelf region block later judgedwrong_region: the lower shelf fills the view. (b) Start of a grasp block that fails withno_grasp_contact. (c) A late block of a native sink success whose program had an earlier tool_path_blocked failure. Frames are unmodified 256-pixel trace keyframes. 20

Instant research discovery