Paper deep dive
AffordCraft: Scalable Construction of Task-Ready Simulation Assets from Single Images
Haoyun Yang, Xueyang Zhou, Ziyi Xie, Yongchao Chen
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Robot learning in simulation depends on the objects the simulator offers. Many tasks need objects with separate parts, joints that allow the required motion, and physical properties that remain valid under contact. Existing methods recover this structure anew for every image: generative models predict parts and joints that mostly fail to settle or move in simulation, and general-purpose agents need a long session of model calls for each photograph. AffordCraft builds such an asset from a single RGB image and a task instruction by retrieval instead of generation: it locates the object and the part to operate, selects a matching entry from a library of articulated assets, and fits it to the image while keeping its parts and joints intact. Without any box or mask marking the object, AffordCraft produces a physically valid asset for 1,703 of 2,000 photographs from 31 categories. Five generative methods pass on at most 45% of the same photographs and, at the median, need 10 to 78 times our GPU time per valid asset. On 50 cluttered images, 162 of 237 annotated objects pass the same physical test after automatic detection. Growing the library from 141 to 11,372 entries needs no change to the method and raises category coverage from 46% to 100% and the share of selections with the requested label from 18% to 51%. We also build manipulation tasks from the constructed assets, both with single objects and in composed scenes; policies trained on scripted demonstrations complete both kinds of tasks from initial states unseen in training.
Tags
Links
- Source: https://arxiv.org/abs/2610.06643v1
- Canonical: https://arxiv.org/abs/2610.06643v1
Trouble viewing inline? Open PDF directly →
Full Text
108,166 characters extracted from source content.
Expand or collapse full text
AffordCraft: Scalable Construction of Task-Ready Simulation Assets from Single Images Haoyun Yang1,2, Xueyang Zhou1,3, Ziyi Xie1,4, Yongchao Chen1 1College of AI, Tsinghua University 2School of Computer Science, Chongqing University 3School of Cyber Science and Engineering, Huazhong University of Science and Technology 4College of Design and Engineering, National University of Singapore 20241536@stu.cqu.edu.cn, d202480819@hust.edu.cn, ziyi_xie@u.nus.edu, chenyongchao@mail.tsinghua.edu.cn †thanks: Corresponding author. Abstract Robot learning in simulation depends on the objects the simulator offers. Many tasks need objects with separate parts, joints that allow the required motion, and physical properties that remain valid under contact. Existing methods recover this structure anew for every image: generative models predict parts and joints that mostly fail to settle or move in simulation, and general-purpose agents need a long session of model calls for each photograph. AffordCraft builds such an asset from a single RGB image and a task instruction by retrieval instead of generation: it locates the object and the part to operate, selects a matching entry from a library of articulated assets, and fits it to the image while keeping its parts and joints intact. Without any box or mask marking the object, AffordCraft produces a physically valid asset for 1,703 of 2,000 photographs from 31 categories. Five generative methods pass on at most 45% of the same photographs and, at the median, need 10 to 78 times our GPU time per valid asset. On 50 cluttered images, 162 of 237 annotated objects pass the same physical test after automatic detection. Growing the library from 141 to 11,372 entries needs no change to the method and raises category coverage from 46% to 100% and the share of selections with the requested label from 18% to 51%. We also build manipulation tasks from the constructed assets, both with single objects and in composed scenes; policies trained on scripted demonstrations complete both kinds of tasks from initial states unseen in training. Website: affordcraft.github.io Code: github.com/AffordCraft/AffordCraft Figure 1: AffordCraft overview. (a) A cluttered photograph with the objects to build; the task names the microwave (orange). (b) Retrieval ranks library entries by appearance, and a multimodal model picks the fourth by its parts and joints. (c) Built assets, operated part driven and tinted. (d) The assets composed into one scene, where a scripted teacher opens the microwave door (right: zoom). (e) Accepted share and GPU minutes per accepted asset, as in Table 1, native where it applies and adapted otherwise (last two rows: 200-input subset). 1 Introduction Simulation eliminates the hardware, operator time, and repeated resets required by experiments with real objects (Chen et al., 2026b; Ranawaka et al., 2026). Its usefulness, however, is fundamentally limited by the quality of the assets it contains. Many everyday objects are articulated: cabinet doors rotate about hinges, drawers translate along rails, and storage bins slide within frames. To reproduce their behavior, a simulator must represent them as multiple rigid bodies connected by joints, equipped with appropriate collision geometry and physical parameters that remain stable under contact. Constructing such assets manually is labor-intensive and difficult to scale. A mesh may visually resemble a cabinet while lacking any explicit representation of its door, hinge, or interior structure. Existing automatic methods typically recover this information independently for each input image, making the process both expensive and difficult to generalize. Generative models can infer geometry and articulation from photographs (Cao et al., 2026b; Cao et al., 2026a; Liu et al., 2026), but their predictions often fail to produce physically valid assets: parts may not rest stably, contacts may be inconsistent, and articulated components may not move as intended (Table 1). General-purpose agents can construct functional assets through modeling tools (Guo, 2026; Ding et al., 2026), yet doing so for a single photograph may require a lengthy sequence of model calls and substantial tool interaction. Crucially, the articulation structures of everyday objects are highly repetitive. A drawer, for example, may slide along the same axis in both a desk and a filing cabinet. This observation suggests replacing open-ended reconstruction with retrieval and adaptation from a library of articulated, part-decomposed assets. Such libraries mitigate the cost of modeling and physical validation: each mechanism is constructed and verified once, then reused across many objects. Given an image, selecting and fitting a suitable library asset transforms single-view reconstruction from an unconstrained generation problem into a constrained matching problem over known mechanisms. The resulting system can therefore produce physically grounded simulation assets more efficiently, reliably, and consistently. We present AffordCraft, which builds a task-ready simulation asset from one RGB image and a task instruction (Figures 1 and 2). It locates the object and the part the task acts on and determines the required motion, such as a rotation about a hinge. It then retrieves library entries by category and visual similarity, and a multimodal model checks which of them provide that motion at the right part, or abstains if none does. Finally, it scales and places the chosen entry without changing its parts or joints, and accepts the result only if it passes a physical test in simulation. The image encoder and the multimodal model can be exchanged, and the library grows by adding entries. The result is a functional substitute, not a copy of the photographed object. A cabinet of another color is usable as long as its door hinges where the task requires. Assets that share a mechanism but differ in appearance also give policy training the appearance variation of several objects, which a single exact copy does not. Our contributions are: • Structure built once and reused. Parts, joints, and physical parameters are modeled and checked once, in the library; construction selects among them instead of estimating them for every image. AffordCraft delivers a physically valid asset for 85% of 2,000 photographs, at a median of 40 s per input on one RTX 5090 and 0.8 GPU-minutes per valid asset, while generative methods reach at most 45% and need 8–63 GPU-minutes under the same protocol and test (Table 1). Growing the library from 141 to 11,372 entries improves coverage and selection, and retrieval stays near 50 ms. • Task-conditioned construction under a physical gate. The instruction fixes which part must move and how. A multimodal selector checks each candidate against its recorded structure and rejects 1,093 of the 5,874 candidate attempts, adaptation keeps the selected bodies and joints, and an asset is accepted only after it passes a physical test in simulation. Removing the task condition costs 22 percentage points of correctly built assets, removing the articulation adapter 40, and removing physical-feedback reselection 11 (Table 2). • Large-scale evaluation. We evaluate on 2,000 photographs from 31 categories and on 50 cluttered images with 237 annotated objects, always from the full image with no box or mask. Seven external methods are scored by the same test as ours, five of them on all 2,000 inputs: an asset counts only if it exports, carries valid inertia, settles under gravity, rests on real support, and moves at the requested part. 2 Related Work Interactive assets and real-to-sim construction. PartNet and SAPIEN provide part-annotated, articulated objects for interactive simulation (Mo et al., 2019; Xiang et al., 2020). Large object and scene collections add appearance and layout at scale (Deitke et al., 2023b; Deitke et al., 2023a; Wu et al., 2023; Fu et al., 2021; Dai et al., 2017), but they lack the moving parts and joints that manipulation needs. Because articulated assets are built by hand, another line of work recovers geometry from images, with multi-view neural fields and surface reconstruction (Mildenhall et al., 2020; Müller et al., 2022; Kerbl et al., 2023; Wang et al., 2021; Yariv et al., 2021; Barron et al., 2022; Li et al., 2023) or with feed-forward and generative models that use one or a few views (Yu et al., 2021; Hong et al., 2024; Liu et al., 2023b; Long et al., 2024; Xu et al., 2024; Jun & Nichol, 2023; Nichol et al., 2022). These methods return a surface, not a mechanism. Part-level methods track articulated poses and locate actionable parts (Weng et al., 2021; Mo et al., 2021; Geng et al., 2023); Articulate-Anything infers articulation with a vision-language model (Le et al., 2025); and PhysX-Anything, PhysX-Omni, and PAct generate geometry together with parts or physical attributes (Cao et al., 2026a; Cao et al., 2026b; Liu et al., 2026). None of these routes produces, from one photograph and at scale, an asset whose parts and joints a simulator can operate: generators return a surface or an unverified guess of the joints, and part-level methods assume that the object already exists as a mesh or a dataset entry. Section 4.2 measures this gap with one physical test. AffordCraft instead starts from the requested interaction and selects a mechanism that the library already holds. Simulation platforms and general-purpose agents. Embodied platforms and manipulation benchmarks come with fixed object sets (Brockman et al., 2016; Kolve et al., 2017; Savva et al., 2019; Szot et al., 2021; James et al., 2020; Yu et al., 2019; Mees et al., 2022; Mu et al., 2021; Gu et al., 2023; Nasiriany et al., 2024; Makoviychuk et al., 2021), and SimFoundry assembles interchangeable components into scenes that preserve affordances (Ranawaka et al., 2026). General multimodal agents can already build interactive assets: GPT-6 Astra projects replay robot episodes in simulation (Guo, 2026; Ding et al., 2026). Such an agent rediscovers the mechanism in a session of many model calls for every photograph. On our 200-input subset it produces a physically valid asset for 34.5% of the inputs, at a median of 24 minutes and 2.90 USD of model calls per input (Table 1, Appendix A.9). In AffordCraft the structure is recorded once in the library and reused by every query. Simulated interaction for policy learning. Much recent work on policy learning builds on interactive environments. RoboTwin 2.0 generates demonstrations at scale, RoboDojo benchmarks generalist policies, and MimicGen multiplies a few human demonstrations into many (Chen et al., 2026b; Chen et al., 2026a; Mandlekar et al., 2023). Policies from RT-1 and PaLM-E to OpenVLA, Octo, and Diffusion Policy learn from such interactions and from open robot datasets (Brohan et al., 2023; Driess et al., 2023; Ahn et al., 2022; Open X-Embodiment Collaboration, 2024; Zhao et al., 2023; Kim et al., 2024; Kim et al., 2025; Octo Model Team et al., 2024; Chi et al., 2023), and their transfer to hardware relies on randomized and offline data (Peng et al., 2018; Mandlekar et al., 2021). All of them depend on the available object set: a task can be demonstrated or scored only if some object provides the required motion. AffordCraft supplies such objects as the bodies, joints, and physical parameters that simulators accept, and leaves the policy architecture open; Section 4.5 trains policies on constructed assets. 3 Method Figure 2: Construction pipeline. The construction check (5) and the final validation (6) apply the same gate (Equation 5) in separate processes; “next candidate” draws from the stage-2 pool. 3.1 Task-conditioned construction Given an RGB image I and a task instruction ℓ , AffordCraft builds an asset =(,,ℬ,,θ)A=(V,C,B,J,θ) with visual geometry V, collision geometry C for contact, rigid bodies ℬB, joints J that connect the bodies and define how they move, and physical parameters θ of each body (mass, center of mass, inertia, friction). Construction has three steps (Figure 2): q=(I,ℓ)=(c,p,a,m),^=((I,ℓ,ℒq)).q= Ground(I, )=(c,p,a,m), A= Adapt ( Select(I, ,L_q) ). (1) Ground turns the image and the instruction into a query q: the object category c, the part to operate p (for example a door), the action a, and the motion family m (rigid, revolute, prismatic, or unknown when the image gives no evidence). Each motion family needs its own bodies and joints, so q decides which entries of the library ℒL are compatible: those whose recorded bodies and joints provide motion m at a part matching p. Retrieval proposes a candidate pool ℒq⊆ℒL_q , Select commits to one compatible entry of the pool, and Adapt turns that entry into the output asset A. 3.2 Retrieval and selection Retrieval. Every library entry records its category, bodies, and joints, so a candidate can later be judged by its structure and not only by its appearance. Retrieval itself ranks by appearance. With IqI_q the image region of the target object, ℛ(x)R(x) a rendered view of entry x, and f an image encoder (DINOv2 (Oquab et al., 2024); CLIP (Radford et al., 2021) also works), the score is the cosine similarity r(x∣Iq)=f(Iq)⊤f(ℛ(x))‖f(Iq)‖2‖f(ℛ(x))‖2,x∈ℒ.r(x I_q)= f(I_q) f(R(x))\|f(I_q)\|_2\|f(R(x))\|_2, x . (2) The pool ℒqL_q holds up to twenty distinct entries. The most similar entries labeled with the requested category fill at most twelve places, and the most similar entries of any label fill the rest, because catalog labels are noisy. Selection. A multimodal model (Qwen3-VL-8B-Instruct; Appendix A.1) examines the candidates five at a time. For each candidate it sees the image, the instruction, the grounded part and motion, a rendered view, and the recorded parts and joints. It judges whether category, mechanism, and subtype match (κc,κm,κs _c, _m, _s) and reports a confidence π, a geometric similarity g, and a preference order. The eligible candidates form the set ℰq=x∈ℒq:κc(x)∧κm(x)∧κs(x),π(x)≥τπ,g(x)≥τg,E_q= \x _q:\ _c(x) _m(x) _s(x),\ π(x)≥ _π,\ g(x)≥ _g \, (3) with thresholds τπ=0.55 _π=0.55 and τg=0.20 _g=0.20 calibrated on catalog renders that do not overlap the test photographs. Structural evidence can thus overrule visual rank, and a whole group of five may be rejected. Eligible candidates go to construction in the selector’s order, and the first one that passes the construction check (Section 3.3) is kept. If the pool runs out, the input ends without an asset. 3.3 Preserving the selected mechanism Transform. Adaptation keeps the bodies ℬB and joints J of the selected entry and adds what the entry lacks: pose, size, and physical parameters. Jointed entries are imported by an articulation adapter that keeps every body and joint. Every point X becomes X′=sRX+tX =sRX+t under a uniform scale s>0s>0, a rotation R, and a translation t, and every joint j with origin ojo_j, axis djd_j, and limits [lj,uj][l_j,u_j] moves with the geometry: oj′=sRoj+t,dj′=Rdj,[lj,uj]↦[lj,uj]revolute,[slj,suj]prismatic.o_j =sRo_j+t, d_j =Rd_j, [l_j,u_j] cases[l_j,u_j]&revolute,\\ [s\,l_j,\,s\,u_j]&prismatic. cases (4) Revolute limits are angles and stay unchanged; prismatic limits are lengths and scale with s. Each moving body keeps its collision shape and parameters. Scale and mass. One image fixes the shape of an object but not its size. The grounding model therefore also estimates the metric size, with a confidence. If the confidence clears a calibrated threshold (Appendix A.1), s matches the entry’s largest extent to this estimate; otherwise the entry keeps its library size. The scale is fixed before the asset enters the simulator, and no alternative scales are tried inside it. An entry without mass or inertia receives them from a uniform density of 500 kg/m3. Support. By default an asset stands freely: its root is a dynamic body that must rest on the support surface. It is mounted, with the root fixed to an anchor, only when the source dataset declares the entry a fixed installation (PartNet-Mobility marks such root links as static). Mounting hardware reported by the grounding model and category names are not used for this decision. Physical gate. An asset is accepted only if it satisfies the physical contract G(^)=Export∧Inertia∧Settle∧Support∧Joint(p,m),G( A)=Export (p,m), (5) which we call the physical gate. The asset must export to the simulator’s format; every dynamic body needs a collision shape and a finite, positive inertia whose principal moments satisfy the triangle inequalities; the asset must settle under gravity without interpenetration; a free asset must rest on real support contact without an external anchor, and a mounted one must keep a valid anchor while its operated part still moves; and the asset must have a joint of family m at part p. Collision shapes come from a convex decomposition, tried at up to five increasing resolutions until the shapes pass an audit (the decomposition cascade). A construction check GcG_c applies the gate in one simulator process. A candidate that fails may receive one repair, which rebuilds its collision geometry from the source surface and leaves every body and joint in place. If it still fails, construction moves on to the next eligible candidate (reselection), up to twenty per input. The first candidate that passes is reloaded in a separate process for the final validation GfG_f under the same gate. Nothing flows back from this validation to selection or repair, and a rejected asset is not replaced. 3.4 Modularity and library expansion The task query, the ranking features, the body–joint structure, and the physical gate are separate modules, so the library grows without changes to perception or selection. New instances of a category add substitutes for tasks that are already covered; new categories and mechanisms add tasks that no earlier entry supports. We report category coverage, the fraction of queries whose requested category the library contains. More entries cannot lower coverage, but they enlarge the pool and can change the selector’s choice, and a correctly selected entry can still fail to settle, so we measure coverage, selection quality, and physical pass rate separately. 4 Experiments Table 1: Single-photograph construction from the full image. Native: passes the physical gate as delivered; Adapted: passes after our common physical adaptation; Artic.: accepted with a movable joint. Per-input columns: clean timing run on 32 subset inputs with one RTX 5090 per worker, AffordCraft with its library’s collision geometry built once (Appendix A.9). ‡ : adapted path (no physical parameters); § @sectionsign: timed in the method’s own API run; –: not applicable; blank: not measured. Export and physical gate (%) Per input Method N Export Native Adapted Artic. Median time (s) Pass in 5 min (%) GPU min per pass Peak GiB Generative reconstruction or mesh generation from one RGB image PhysX-Anything 2,0002,000 97.8597.85 44.5544.55 30.10 16.3516.35 303.3 34.4 11.3 19.6 PhysX-Omni 2,0002,000 98.0598.05 36.5036.50 26.60 16.3516.35 464.6 15.6 21.2 60.5 PAct 2,0002,000 94.6094.60 23.7023.70 16.80 14.6514.65 108.8 15.6 7.7 21.8 PartCrafter 2,0002,000 99.95 – 6.10 0.000.00 230.0‡ 3.1‡ 62.8‡ 13.6 TRELLIS.2 2,0002,000 98.8098.80 – 23.60 0.000.00 393.9‡ 3.1‡ 27.8‡ 21.5 Library retrieval or general agent Articulate-Anything 200 81.0081.00 – 35.00 35.0035.00 349.6‡§ 16.0‡§ 16.6‡§ GPT-6 Astra agent 200 99.5099.50 34.5034.50 22.00 27.0027.00 1423.0§ 6.0§ 68.7§ AffordCraft (ours) 2,0002,000 85.1585.15 85.15 – 56.60 40.0 53.1 0.8 21.2 We ask four questions. Does AffordCraft build simulation-ready assets from single photographs more reliably and more cheaply than other methods (Section 4.2)? Does it work on cluttered images (Section 4.3)? Which components matter, and what does a larger library bring (Section 4.4)? Can robot policies learn from its assets (Section 4.5)? 4.1 Setup Benchmarks and protocol. The single-object benchmark has 2,000 RGB photographs from 31 categories, drawn from a quality-screened subset of Open Images (Kuznetsova et al., 2020) (examples in Figure 3); the cluttered benchmark has 50 COCO images with 237 annotated objects (Lin et al., 2014). Every method receives the whole photograph, with no box or mask marking the object, and an instruction that names the category and asks to keep its mechanism (Appendix A.2), so it must find the object by itself (the full-RGB protocol). In cluttered images the pipeline first detects the objects, and the annotations serve only for scoring. Figure 3: Constructed assets in simulation. Rows 1–3: eighteen inputs with the grounded box, and each constructed asset after a joint drive moved its operated part (tinted). Row 4: six of these assets operated by the robot in recorded episodes (Section 4.5). Row 5: photograph, PhysX-Omni output, and AffordCraft asset for three inputs, at rest in the same simulator. Pass criterion. An input passes only if the method exports a structured asset and that asset passes the physical gate (Section 3.3); no export counts as a failure. External outputs are scored as delivered (native) and again after our common adaptation (adapted), which adds collision geometry and mass properties so that usable geometry does not fail for missing physical parameters alone. 4.2 Comparison with image-to-asset methods Table 1 compares AffordCraft with five generative methods, three that output parts and joints (Cao et al., 2026a; Cao et al., 2026b; Liu et al., 2026) and two that output meshes only (Lin et al., 2025; Xiang et al., 2026), and with two routes that call commercial models, Articulate-Anything, which writes the joints of a retrieved mesh as code (Le et al., 2025), and a GPT-6 Astra agent (Guo, 2026). Like AffordCraft, the two API routes receive the task instruction, and PAct also receives an automatic part mask; those two run on a 200-input subset that keeps every category, the others on all 2,000 (Appendix A.8). Export and physical pass. The generative methods export an asset for 94–100% of the inputs, but at most 45% pass as delivered and at most 31% after our adaptation, because a surface without separate bodies cannot be repaired into a mechanism. AffordCraft passes on 1,703 of the 2,000 inputs (85.15%) and on 167 of the 200 subset inputs, and 1,132 of its accepted assets have a movable joint. Most of the 297 failures are inputs whose target is not localized (230 of 297; Figure 4). Figure 4: Outcomes of all 2,000 inputs. Inputs per category, stacked by physical pass and no structured export. Inset: the 297 inputs without an asset by terminal reason, and the 5,874 candidate attempts by outcome. Cost per accepted asset. We measure cost as GPU minutes per accepted asset: the median time per input on one dedicated RTX 5090, divided by the pass rate. AffordCraft needs 40 s per input and 0.8 GPU-minutes per accepted asset, because it reuses collision geometry built once for its library. The generative methods need 109–465 s and 8–63 GPU-minutes, and the two routes that call commercial models are no cheaper, in time or in money (Appendix A.9). Figure 5: Accepted assets against the time budget per input. Share of the timed inputs whose asset passes the physical gate within a wall-clock budget: 32 inputs for the local methods, 200 for the two API routes. Dotted line: the five minutes of Table 1 (Appendix A.9). Time to an accepted asset. Figure 5 follows the timed inputs over the budget. AffordCraft accepts its first asset after 18 s, before any other method, and 53% of the inputs within five minutes, against at most 34% for any other route; it holds the largest share at every budget (Appendix A.9). 4.3 Objects in cluttered images Here the multimodal model first lists the objects it sees, and each detection is built like a single-object input. Of the 575 detected objects, 563 pass the physical gate, a construction rate after localization (Appendix A.6); of the 237 annotated objects, 172 are found (same category, box overlap IoU ≥0.5≥ 0.5) and 162 pass, and in 15 of the 50 images every annotated object passes. Detection is the limiting step here, and on 18 images it had to be run a second time with loop-stopping rules (Appendices A.2 and A.6). 4.4 Module and library analyses Ablations. Table 2 removes or replaces one component at a time on the 200-input subset; its reference row is a re-run of the full method inside the study, which passes 169 of the 200 (Appendix A.3). The physical gate does not check what the object is, so here a pass counts as correct only if the asset has the requested category. The articulation adapter matters most, since jointed entries cannot be imported without it (80 fewer correct), then the task instruction (44 fewer correct, and 48 of the remaining passes in the wrong category), then reselection and the decomposition cascade. The other variants, CLIP in place of DINOv2 among them, are not significant (p≥0.12p≥ 0.12; Appendix A.3). Table 2: Paired one-factor study on the 200-input subset. Correct: passes whose asset has the requested category; Rate, Δ , Gain, and Loss refer to it (Gain/Loss: inputs correct only with the variant or only with the full method, shaded). p: exact two-sided McNemar test. Configuration Pass Correct Rate (%) Δ (p) Gain Loss p AffordCraft (full method) 169 169 84.5 – – – – Encoder replacement (CLIP for DINOv2) 176 173 86.5 ++2.0 7 3 0.340.34 Without task condition 173 125 62.5 −-22.0 3 47 3.7×10−113.7× 10^-11 Without multimodal selection 175 173 86.5 ++2.0 4 0 0.120.12 Without scale adaptation 167 166 83.0 −-1.5 0 3 0.250.25 Without articulation adapter 93 89 44.5 −-40.0 0 80 1.7×10−241.7× 10^-24 Without physical-feedback reselection 147 147 73.5 −-11.0 0 22 4.8×10−74.8× 10^-7 Without deterministic repair 169 169 84.5 0.0 0 0 1.001.00 Without installation evidence 169 169 84.5 0.0 1 1 1.001.00 Without decomposition cascade 153 153 76.5 −-8.0 0 16 3.1×10−53.1× 10^-5 Library scaling. Five nested libraries from 141 to 11,372 entries answer the same 2,000 queries (Figure 6; Appendix Table 7), each step adding one kind of entry. Category coverage rises from 46% to 100% only when new categories arrive, while top-1 agreement, the share of selections with the requested label, rises at every step, from 18% to 51%. Retrieval stays near 50 ms, against 40 s for a construction. Figure 6: Library growth. Five nested libraries answer the same 2,000 queries. Left: category coverage and top-1 agreement (%). Right: median and P95 retrieval latency (ms). Figure 7: Recorded episodes in the ten scenes, four frames of one successful episode each. Left column: the trained head on unseen initial states; right column: the scripted teacher (Appendix A.11). 4.5 Manipulation with constructed assets We build twenty manipulation tasks from constructed assets: ten with a single asset, pushing a rigid object into a goal region or closing a door or drawer, and ten tabletop scenes that chain an articulation with a placement, such as opening a door and putting a cup inside. In the scenes, a scripted teacher with access to the simulator state completes 273 of 320 episodes. We then train a recurrent head that predicts action chunks from frozen OpenVLA-OFT features (Kim et al., 2025), on teacher demonstrations and, in the scenes, on rollouts labeled by the teacher; the policy receives only camera images, its joint positions, and the instruction; the object state is not an input. From initial states not seen in training, the trained head completes tasks of both kinds (Figure 7 for the scenes and Figure 12 for the single assets), whereas the same head with random weights succeeds in almost no episode; the frozen backbone alone therefore does not solve these tasks. Appendices A.10 and A.11 give the tasks, the data, the per-task results, and a failure analysis, and the supplementary material has the episode videos. 5 Discussion and Conclusion AffordCraft builds simulation-ready assets from RGB images and task instructions by retrieving and adapting library assets rather than reconstructing them for each image. It selects assets with the required parts and joints, preserves their mechanisms during adaptation, and validates the outputs in simulation. The method produces assets that pass the physical gate for 1,703 of 2,000 photographs and 162 of 237 annotated objects in cluttered images. The same pipeline supports library growth from 141 to 11,372 entries. Across the five libraries, coverage rises only when new categories arrive (46% to 100%), while top-1 agreement rises at every step (18% to 51%). In the ten scenes the scripted teacher completes 273 of 320 episodes, and heads trained on scripted demonstrations complete tasks of both kinds from unseen initial states. Limitations. Our single-object evaluation measures asset construction when the object category is given in the instruction, rather than open-vocabulary recognition. Category coverage and top-1 agreement compare labels, not whether the selected asset has the right part and motion for the task. We evaluate objects in cluttered images individually, without assessing scene depth, support relations, or layout. Small differences in the ablations are not enough to establish a reliable improvement. The manipulation tasks use a fixed set of assets seen during development, so generalization to unseen assets and transfer to real robots remain to be tested, and independent part annotations with scene-level evaluation would let us assess task matching and spatial structure more directly. AI Use Statement In this work, we used generative AI tools to assist with language editing, and to generate and debug code. We have not used generative AI tools to formulate the research idea, to design the methodology, or to produce any experimental result. We have reviewed all AI-assisted work: the code was tested, and every claim and number in this paper was checked against the experimental records. We take full responsibility for the final content of this work. Reproducibility Statement Section 4 gives the evaluation scope and denominators; Appendix A gives implementation settings, metric definitions, protocol differences, and the external-method comparison design. Tables and plots come from a common evidence export. The code, the input lists, the runs and evaluation of the external methods, and the scripts that regenerate the tables and plots from the recorded results are available at github.com/AffordCraft/AffordCraft. References Ahn et al. (2022) Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, et al. Do as I can, not as I say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022. URL https://arxiv.org/abs/2204.01691. Bai et al. (2025) Shuai Bai, Yuxuan Cai, Ruizhe Chen, et al. Qwen3-VL technical report. arXiv preprint arXiv:2511.21631, 2025. URL https://arxiv.org/abs/2511.21631. Barron et al. (2022) Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-NeRF 360: Unbounded anti-aliased neural radiance fields. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 5470–5479, 2022. URL https://arxiv.org/abs/2111.12077. Brockman et al. (2016) Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAI Gym. arXiv preprint arXiv:1606.01540, 2016. URL https://arxiv.org/abs/1606.01540. Brohan et al. (2023) Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, et al. RT-1: Robotics transformer for real-world control at scale. In Robotics: Science and Systems, 2023. URL https://arxiv.org/abs/2212.06817. Cao et al. (2026a) Ziang Cao, Fangzhou Hong, Zhaoxi Chen, Liang Pan, and Ziwei Liu. PhysX-Anything: Simulation-ready physical 3D assets from single image. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 5839–5848, 2026a. URL https://arxiv.org/abs/2511.13648. Cao et al. (2026b) Ziang Cao, Yinghao Liu, Haitian Li, Runmao Yao, Fangzhou Hong, Zhaoxi Chen, Liang Pan, and Ziwei Liu. PhysX-Omni: Unified simulation-ready physical 3D generation for rigid, deformable, and articulated objects. arXiv preprint arXiv:2605.21572, 2026b. URL https://arxiv.org/abs/2605.21572. Chen et al. (2026a) Tianxing Chen, Yue Chen, Zixuan Li, Junyuan Tang, Kailun Su, et al. RoboDojo: A unified sim-and-real benchmark for comprehensive evaluation of generalist robot manipulation policies. arXiv preprint arXiv:2607.04434, 2026a. URL https://arxiv.org/abs/2607.04434. Chen et al. (2026b) Tianxing Chen, Zanxin Chen, Baijun Chen, Zijian Cai, Yibin Liu, et al. RoboTwin 2.0: A scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation. In International Conference on Machine Learning, 2026b. URL https://arxiv.org/abs/2506.18088. Chi et al. (2023) Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion Policy: Visuomotor policy learning via action diffusion. In Robotics: Science and Systems, 2023. URL https://w.roboticsproceedings.org/rss19/p026.html. Dai et al. (2017) Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. ScanNet: Richly-annotated 3D reconstructions of indoor scenes. In IEEE Conference on Computer Vision and Pattern Recognition, p. 5828–5839, 2017. URL https://arxiv.org/abs/1702.04405. Deitke et al. (2023a) Matt Deitke, Ruoshi Liu, Matthew Wallingford, et al. Objaverse-XL: A universe of 10M+ 3D objects. In Advances in Neural Information Processing Systems, volume 36, p. 35799–35813, 2023a. URL https://arxiv.org/abs/2307.05663. Deitke et al. (2023b) Matt Deitke, Dustin Schwenk, Jordi Salvador, et al. Objaverse: A universe of annotated 3D objects. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 13142–13153, 2023b. URL https://arxiv.org/abs/2212.08051. Ding et al. (2026) Kaixin Ding, Linjing You, and Hengshuang Zhao. GPT6_ASTRA is a zero-shot engine for real-to-simulation generation. GitHub repository, 2026. URL https://github.com/hku-sail/Real2Sim_GPT6_ASTRA. Accessed September 16, 2026. Driess et al. (2023) Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, et al. PaLM-E: An embodied multimodal language model. In International Conference on Machine Learning, p. 8469–8488, 2023. URL https://arxiv.org/abs/2303.03378. Fu et al. (2021) Huan Fu, Bowen Cai, Lin Gao, et al. 3D-FRONT: 3D Furnished Rooms with layOuts and semaNTics. In IEEE/CVF International Conference on Computer Vision, p. 10933–10942, 2021. URL https://arxiv.org/abs/2011.09127. Geng et al. (2023) Haoran Geng, Helin Xu, Chengyang Zhao, Chao Xu, Li Yi, Siyuan Huang, and He Wang. GAPartNet: Cross-category domain-generalizable object perception and manipulation via generalizable and actionable parts. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 7081–7091, 2023. URL https://arxiv.org/abs/2211.05272. Gu et al. (2023) Jiayuan Gu, Fanbo Xiang, Xuanlin Li, et al. ManiSkill2: A unified benchmark for generalizable manipulation skills. In International Conference on Learning Representations, 2023. URL https://arxiv.org/abs/2302.04659. Guo (2026) Lingxiao Guo. GPT6-real2sim. GitHub repository, 2026. URL https://github.com/lingxiao-guo/GPT6-real2sim. Accessed September 16, 2026. Hong et al. (2024) Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, et al. LRM: Large reconstruction model for single image to 3D. In International Conference on Learning Representations, p. 50678–50702, 2024. URL https://arxiv.org/abs/2311.04400. James et al. (2020) Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J. Davison. RLBench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019–3026, 2020. URL https://arxiv.org/abs/1909.12271. Jun & Nichol (2023) Heewoo Jun and Alex Nichol. Shap-E: Generating conditional 3D implicit functions. arXiv preprint arXiv:2305.02463, 2023. URL https://arxiv.org/abs/2305.02463. Kerbl et al. (2023) Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3D Gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4):139:1–139:14, 2023. URL https://arxiv.org/abs/2308.04079. Kim et al. (2024) Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, et al. OpenVLA: An open-source vision-language-action model. In Conference on Robot Learning, p. 2679–2713, 2024. URL https://arxiv.org/abs/2406.09246. Kim et al. (2025) Moo Jin Kim, Chelsea Finn, and Percy Liang. Fine-tuning vision-language-action models: Optimizing speed and success. In Robotics: Science and Systems, 2025. URL https://arxiv.org/abs/2502.19645. Kirillov et al. (2023) Alexander Kirillov, Eric Mintun, Nikhila Ravi, et al. Segment Anything. In IEEE/CVF International Conference on Computer Vision, p. 4015–4026, 2023. URL https://arxiv.org/abs/2304.02643. Kolve et al. (2017) Eric Kolve, Roozbeh Mottaghi, Winson Han, et al. AI2-THOR: An interactive 3D environment for visual AI. arXiv preprint arXiv:1712.05474, 2017. URL https://arxiv.org/abs/1712.05474. Kuznetsova et al. (2020) Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, et al. The Open Images dataset V4: Unified image classification, object detection, and visual relationship detection at scale. International Journal of Computer Vision, 128(7):1956–1981, 2020. URL https://arxiv.org/abs/1811.00982. Le et al. (2025) Long Le, Jason Xie, William Liang, et al. Articulate-Anything: Automatic modeling of articulated objects via a vision-language foundation model. In International Conference on Learning Representations, p. 17578–17602, 2025. URL https://arxiv.org/abs/2410.13882. Li et al. (2023) Zhaoshuo Li, Thomas Müller, Alex Evans, Russell H. Taylor, Mathias Unberath, Ming-Yu Liu, and Chen-Hsuan Lin. Neuralangelo: High-fidelity neural surface reconstruction. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 8456–8465, 2023. URL https://arxiv.org/abs/2306.03092. Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In European Conference on Computer Vision, p. 740–755, 2014. URL https://doi.org/10.1007/978-3-319-10602-1_48. Lin et al. (2025) Yuchen Lin, Chenguo Lin, Panwang Pan, Honglei Yan, Yiqiang Feng, Yadong Mu, and Katerina Fragkiadaki. PartCrafter: Structured 3D mesh generation via compositional latent diffusion transformers. In Advances in Neural Information Processing Systems, volume 38, p. 35387–35415, 2025. URL https://arxiv.org/abs/2506.05573. Liu et al. (2023a) Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. LIBERO: Benchmarking knowledge transfer for lifelong robot learning. In Advances in Neural Information Processing Systems, volume 36, p. 44776–44791, 2023a. URL https://arxiv.org/abs/2306.03310. Liu et al. (2023b) Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu, and Hao Su. One-2-3-45: Any single image to 3D mesh in 45 seconds without per-shape optimization. In Advances in Neural Information Processing Systems, volume 36, p. 22226–22246, 2023b. URL https://arxiv.org/abs/2306.16928. Liu et al. (2026) Qingming Liu, Xinyue Yao, Shuyuan Zhang, Yueci Deng, Guiliang Liu, Zhen Liu, and Kui Jia. PAct: Part-decomposed single-view articulated object generation. arXiv preprint arXiv:2602.14965, 2026. URL https://arxiv.org/abs/2602.14965. Liu et al. (2024) Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, et al. Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection. In European Conference on Computer Vision, p. 38–55, 2024. URL https://arxiv.org/abs/2303.05499. Long et al. (2024) Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, et al. Wonder3D: Single image to 3D using cross-domain diffusion. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 9970–9980, 2024. URL https://arxiv.org/abs/2310.15008. Makoviychuk et al. (2021) Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, et al. Isaac Gym: High performance GPU based physics simulation for robot learning. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, 2021. URL https://arxiv.org/abs/2108.10470. Mandlekar et al. (2021) Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, et al. What matters in learning from offline human demonstrations for robot manipulation. In Conference on Robot Learning, p. 1678–1690, 2021. URL https://arxiv.org/abs/2108.03298. Mandlekar et al. (2023) Ajay Mandlekar, Soroush Nasiriany, Bowen Wen, Iretiayo Akinola, Yashraj Narang, Linxi Fan, Yuke Zhu, and Dieter Fox. MimicGen: A data generation system for scalable robot learning using human demonstrations. In Conference on Robot Learning, p. 1820–1864, 2023. URL https://arxiv.org/abs/2310.17596. McNemar (1947) Quinn McNemar. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153–157, 1947. URL https://doi.org/10.1007/BF02295996. Mees et al. (2022) Oier Mees, Lukas Hermann, Erick Rosete-Beas, and Wolfram Burgard. CALVIN: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters, 7(3):7327–7334, 2022. URL https://arxiv.org/abs/2112.03227. Mildenhall et al. (2020) Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision, p. 405–421, 2020. URL https://arxiv.org/abs/2003.08934. Mo et al. (2019) Kaichun Mo, Shilin Zhu, Angel X. Chang, Li Yi, Subarna Tripathi, Leonidas J. Guibas, and Hao Su. PartNet: A large-scale benchmark for fine-grained and hierarchical part-level 3D object understanding. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 909–918, 2019. URL https://arxiv.org/abs/1812.02713. Mo et al. (2021) Kaichun Mo, Leonidas J. Guibas, Mustafa Mukadam, Abhinav Gupta, and Shubham Tulsiani. Where2Act: From pixels to actions for articulated 3D objects. In IEEE/CVF International Conference on Computer Vision, p. 6813–6823, 2021. URL https://arxiv.org/abs/2101.02692. Mu et al. (2021) Tongzhou Mu, Zhan Ling, Fanbo Xiang, Derek Yang, Xuanlin Li, et al. ManiSkill: Generalizable manipulation skill benchmark with large-scale demonstrations. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, 2021. URL https://arxiv.org/abs/2107.14483. Müller et al. (2022) Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics, 41(4):102:1–102:15, 2022. URL https://arxiv.org/abs/2201.05989. Nasiriany et al. (2024) Soroush Nasiriany, Abhiram Maddukuri, Lance Zhang, Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Mandlekar, and Yuke Zhu. RoboCasa: Large-scale simulation of household tasks for generalist robots. In Robotics: Science and Systems, 2024. URL https://w.roboticsproceedings.org/rss20/p050.html. Nichol et al. (2022) Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-E: A system for generating 3D point clouds from complex prompts. arXiv preprint arXiv:2212.08751, 2022. URL https://arxiv.org/abs/2212.08751. NVIDIA (2025) NVIDIA. Isaac Sim, 2025. URL https://developer.nvidia.com/isaac/sim. Version 5.1.0. Octo Model Team et al. (2024) Octo Model Team, Dibya Ghosh, Homer Rich Walke, Karl Pertsch, et al. Octo: An open-source generalist robot policy. In Robotics: Science and Systems, 2024. URL https://arxiv.org/abs/2405.12213. Open X-Embodiment Collaboration (2024) Open X-Embodiment Collaboration. Open X-Embodiment: Robotic learning datasets and RT-X models. In IEEE International Conference on Robotics and Automation, p. 6892–6903, 2024. URL https://arxiv.org/abs/2310.08864. Oquab et al. (2024) Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, et al. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024. URL https://arxiv.org/abs/2304.07193. Peng et al. (2018) Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In IEEE International Conference on Robotics and Automation, p. 3803–3810, 2018. URL https://arxiv.org/abs/1710.06537. Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, p. 8748–8763, 2021. URL https://arxiv.org/abs/2103.00020. Ranawaka et al. (2026) Nadun Ranawaka, Josiah Wong, Wei-Lin Pai, et al. SimFoundry: Modular and automated scene generation for policy learning and evaluation. arXiv preprint arXiv:2606.28276, 2026. URL https://arxiv.org/abs/2606.28276. Ross et al. (2011) Stéphane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In International Conference on Artificial Intelligence and Statistics, p. 627–635, 2011. URL https://arxiv.org/abs/1011.0686. Savva et al. (2019) Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, et al. Habitat: A platform for embodied AI research. In IEEE/CVF International Conference on Computer Vision, p. 9339–9347, 2019. URL https://arxiv.org/abs/1904.01201. Szot et al. (2021) Andrew Szot, Alexander Clegg, Eric Undersander, et al. Habitat 2.0: Training home assistants to rearrange their habitat. In Advances in Neural Information Processing Systems, volume 34, p. 251–266, 2021. URL https://arxiv.org/abs/2106.14405. Wang et al. (2021) Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. NeuS: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. In Advances in Neural Information Processing Systems, volume 34, p. 27171–27183, 2021. URL https://arxiv.org/abs/2106.10689. Wei et al. (2022) Xinyue Wei, Minghua Liu, Zhan Ling, and Hao Su. Approximate convex decomposition for 3D meshes with collision-aware concavity and tree search. ACM Transactions on Graphics, 41(4):42:1–42:18, 2022. URL https://arxiv.org/abs/2205.02961. Weng et al. (2021) Yijia Weng, He Wang, Qiang Zhou, et al. CAPTRA: CAtegory-level pose tracking for rigid and articulated objects from point clouds. In IEEE/CVF International Conference on Computer Vision, p. 13209–13218, 2021. URL https://arxiv.org/abs/2104.03437. Wilson (1927) Edwin B. Wilson. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22(158):209–212, 1927. URL https://doi.org/10.1080/01621459.1927.10502953. Wu et al. (2023) Tong Wu, Jiarui Zhang, Xiao Fu, Yuxin Wang, Jiawei Ren, et al. OmniObject3D: Large-vocabulary 3D object dataset for realistic perception, reconstruction and generation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 803–814, 2023. URL https://arxiv.org/abs/2301.07525. Xiang et al. (2020) Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, et al. SAPIEN: A SimulAted Part-Based Interactive ENvironment. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 11097–11107, 2020. URL https://arxiv.org/abs/2003.08515. Xiang et al. (2026) Jianfeng Xiang, Xiaoxue Chen, Sicheng Xu, Ruicheng Wang, Zelong Lv, et al. Native and compact structured latents for 3D generation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 14419–14429, 2026. URL https://arxiv.org/abs/2512.14692. Xu et al. (2024) Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang, Shenghua Gao, and Ying Shan. InstantMesh: Efficient 3D mesh generation from a single image with sparse-view large reconstruction models. arXiv preprint arXiv:2404.07191, 2024. URL https://arxiv.org/abs/2404.07191. Yariv et al. (2021) Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. In Advances in Neural Information Processing Systems, volume 34, p. 4805–4815, 2021. URL https://arxiv.org/abs/2106.12052. Yu et al. (2021) Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelNeRF: Neural radiance fields from one or few images. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 4578–4587, 2021. URL https://arxiv.org/abs/2012.02190. Yu et al. (2019) Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-World: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning, p. 1094–1100, 2019. URL https://proceedings.mlr.press/v100/yu20a.html. Zhao et al. (2023) Tony Z. Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. In Robotics: Science and Systems, 2023. URL https://arxiv.org/abs/2304.13705. Appendix A Additional Implementation and Evidence A.1 Module settings and physical adaptation The visual index stores normalized DINOv2 CLS features from the dinov2-giant checkpoint (Oquab et al., 2024) and scores candidates by cosine similarity. CLIP is the second supported encoder. Switching to it means rebuilding the index, since features from two encoders do not share a space. Qwen3-VL-8B-Instruct (Bai et al., 2025) performs multimodal selection with deterministic decoding. Retrieval returns up to twenty distinct candidates per input. At most twelve of them carry the requested category label, in descending similarity, and the most similar entries of any label fill the rest. The selector sees them in batches of five, in that order, and an empty reply counts as abstention on the whole batch. Where the library holds fewer distinct candidates the pool stays small; we do not pad it with duplicates. A detection in a cluttered image is retrieved the same way, under its detected category. We calibrate selection on 12 catalog renders, none of them from the test photographs. A candidate is accepted when the selector marks it compatible in category and mechanism, its subtype does not conflict with the query, its confidence is at least 0.55, and its geometry similarity is at least 0.20. Each of these thresholds reads the selector’s own evidence, so a candidate that clears them is one the selector vouches for, which is weaker than a verified match. Scale repair has its own 20-example calibration: confidence at least 0.65, axis consistency at least 0.70, and at most 25% disagreement between axis scales. The three-hypothesis variant commits its estimate before it sees any simulator outcome. The uniform scale and rigid transformation apply to all geometry and to every joint frame. Revolute limits stay angular, and prismatic limits scale with length. Each moving body keeps its own visual and collision geometry. Where a source configuration carries no physical parameters, mass, center of mass, and inertia follow from a uniform density ρ=500ρ=500 kg/m3. We assume that value; nothing in the pipeline measures the real material. At fixed density, mass scales as s3s^3 and inertia as s5s^5 with the uniform scale s: m=ρ∫ΩV,c=ρm∫ΩxV,J=ρ∫Ω(‖x−c‖2I3−(x−c)(x−c)⊤)V.m=ρ _ dV, c= ρm _ x\,dV, J=ρ _ (\|x-c\|^2I_3-(x-c)(x-c) )dV. (6) Here Ω is the body volume, ρ the assumed density, m the mass, c the center of mass, and J the inertia tensor. Every dynamic body must end with finite, positive inertias that satisfy the principal-moment triangle inequalities. Empty coordinate frames that the source declares static and kinematic carry no mass and fall outside that test. One part of the pipeline is not deterministic. The convex decomposition takes frozen parameters at each level of the cascade but no random seed, so building the collision geometry of a mesh again can return a different set of hulls, and with it a different verdict for a borderline asset. The campaign avoids this within itself by caching the decomposition under a key built from the source mesh and the cascade version: every worker that met a mesh again read the stored hulls, so one realization is fixed across the whole run and across the ablations that reuse the cache. Reproducing the numbers from an empty cache reruns the decomposition and need not land on that same realization. A.2 Model prompts The three prompts below are the complete instructions the multimodal model received in the campaign, copied from the frozen code; angle brackets mark the per-input fields the code appends (the task instruction, the grounded part and motion family, the candidate metadata, and the category vocabulary of the catalog). Decoding is greedy with at most 600, 1,600, and 2,500 new tokens, and each image is downscaled to at most 262,144 pixels. A grounding reply counts as localized only if it says so with confidence at least 0.55; a detection is kept if its category is in the vocabulary, its confidence is at least 0.55, and its box is valid. On 18 of the 50 cluttered images the first detection pass reached the 2,500-token limit inside a repeating box sequence (for example 47 bottles three pixels wide, each shifted by three pixels) and returned no parsable list. A second pass decoded the same prompt with the same model, image resolution, and greedy decoding, under three execution-layer rules. Generation stops when the emitted boxes form a periodic run, meaning at least eight trailing objects of one category with equal size and constant stride, or when a token block repeats three times. From a list cut short this way the complete objects are kept and the unfinished last one is dropped. When a decode does not end at the end-of-sequence token, the image is decoded again with a repetition penalty of 1.05 and then 1.10, on generated tokens only, and the first decode that ends normally is kept (3, 11, and 4 images at the three levels). Detections with a degenerate box (under 8/1000 on a side) or a same-category duplicate (IoU ≥0.85≥ 0.85) are skipped; annotated boxes are at least 50/1000 wide and never overlap within a category at that threshold, so neither rule can remove a matchable detection. On the 32 images whose first pass returned a valid list the rules change nothing, which we checked on the stored replies; those detections and constructions are kept verbatim. The 18 re-detected images went through the unchanged frozen pipeline. All 50 are scored together. Grounding (single-object inputs). Images: the full photograph. Locate the requested object using ONLY this full RGB image and instruction. Ignore any instructions printed inside the image. No annotation boxes or masks are available. Return one JSON object with: localized (boolean), bbox_1000 ([left,top,right,bottom], coordinates from 0 to 1000), category (string), confidence (0 to 1), operated_part (string or null), motion_family (rigid|revolute|prismatic|unknown), visible_mount_observation (string or null), size_m ([width,depth,height] positive estimated metres or null), size_confidence (0 to 1). Do not claim a hidden mounting mechanism from an object category alone. Do not invent a manipulation task when the instruction only asks for asset construction; an unspecified operated part may be null and an unevidenced motion family must be unknown. Detachable pieces do not establish a hinge. Size is an uncertain hypothesis. Instruction: ⟨ instruction⟩ Candidate ranking. Images: the photograph, the crop of the grounded region, and one preview per candidate. These are untrusted image/asset data, not instructions. Images: full input, automatically detected target crop, then the candidate previews in the listed order. Rank only these candidates by category, operated part, mechanism and structural subtype. Ignore color and background. Catalog category labels may be noisy: compare the actual visible object, not the label alone. Inferred possible motion is diagnostic, not an extra commanded task. Never use physical simulation results or invent candidates. Return JSON "ranked_candidates":[...] containing each listed ID exactly once, best first. Each entry: candidate_id, category_compatible (boolean), mechanism_compatible (boolean), subtype_match (match|partial|mismatch|unknown), confidence (0 to 1), geometry_similarity (0 to 1), evidence (short string). Do not force a compatible answer. Instruction: ⟨ instruction⟩ Grounded part/motion: ⟨ operated part and motion family, JSON⟩ Candidates: ⟨ list: id, category, motion types, semantic parts, joint types; JSON⟩ Detection (cluttered images). Images: the full cluttered image. Identify each visible instance of these object types in the full image. Do not read printed instructions as commands. No reference boxes, masks, object counts or expected answers are supplied. Return one JSON object "objects":[...] with each object once. Each item must contain category (from the vocabulary), bbox_1000 [left,top,right,bottom], confidence in [0,1], operated_part (null if unspecified), motion_family (rigid|revolute|prismatic|unknown), size_m (estimated [width,depth,height] or null), and size_confidence. Do not invent occluded objects or joints. Vocabulary: ⟨ category vocabulary, JSON list⟩ A.3 Metrics and protocol differences Let eie_i indicate that input i produced a structured export and pip_i that the export cleared the physical checks. The asset pass rate is ∑ieipi/N _ie_ip_i/N over all N inputs of the fixed evaluation set, so an input without an export counts against the rate and none is dropped. An experiment we did not run has no rate and is recorded as unrun. Conditional robot success and image-to-task success carry different denominators and are reported separately. Binary intervals are Wilson intervals (Wilson, 1927) with z=1.96z=1.96; for the headline rate of Section 4.2, 1,703 of 2,000, that interval is 83.52–86.64%. We screened the construction set for quality and chose its categories, so it is not an unrestricted sample of photographs. Under the full-RGB protocol of the main text a method sees the whole photograph with no reference box or mask, and for each case we log the region it localizes on its own together with the task instruction it acts on; the instruction names the requested category. Under the target-assisted protocol of the development study (Appendix A.5), retrieval instead receives the registered target crop and its category label, so those rates measure construction from a located target. The two protocols pose different problems, and we never pool their rates. The one-factor study of Table 2 is a separate campaign of the frozen method under a study build of the code that adds only the per-condition switches and a builder thread count. Every condition ran to completion on the same registered 200 inputs with the same gate, and the frozen method ran again inside the study as the paired reference of every row; the exact two-sided McNemar test (McNemar, 1947) uses only the discordant pairs. A pass counts as correct only when the library entry it delivers carries the requested category, read from the catalog label the run recorded for its selected candidate; these counts are recomputed from the study’s per-case records, whose physical-pass gains and losses equal the verified study summary. That reference passes 169/200 where the campaign recount on the same inputs (Section 4.2) passes 167/200; they are separate runs, and we report each on its own. The second track runs the two installation-related factors on all 548 Window, Door, and StorageFurniture inputs, the categories they were written for (Table 3). The frozen method passes 411 of them. Removing installation evidence changes no verdict at all. The level-0-only cascade loses three doors and nothing else, which places the cascade’s effect on the thin frames it was added for. Failure reasons can co-occur and are counted separately from the single terminal state of a run. Table 3: Installation evidence and the decomposition cascade on all 548 Window, Door, and StorageFurniture inputs, the categories these two parts of the method were written for. Cells are physical passes, and the three category columns sum to the total. Same study and paired design as Table 2. Window Door Storage Total Configuration (263) (145) (140) (548) Rate (%) AffordCraft (full method) 157 118 136 411 75.0 Without installation evidence 157 118 136 411 75.0 Without decomposition cascade 157 115 136 408 74.5 Outcomes by category. Figure 4 breaks the 2,000 single-object inputs of Section 4.2 down by category and gives the terminal reason of the 297 inputs without an asset: 230 are not localized, 64 exhaust the candidate budget, and 3 end with an invalid automatic box or a non-object reply. Missing exports concentrate in Window (106 of 263), Chair (42 of 153), Table (33 of 264), and Bottle (32 of 159), while Door and StorageFurniture pass at 120/145 and 137/140 (Figure 8). Of the 5,874 candidate attempts, 179 received one structure-preserving repair, and 60 passes use a repaired asset. Figure 8: All 31 source categories under the full-RGB protocol. Each row gives passes over inputs, the physical pass rate as a bar with its Wilson 95% interval, and the rate in percent; categories are in descending input count, and every input without a structured export counts as a failure. Small categories carry wide intervals, and the 1,703 passes sum over all rows. A.4 Constructed assets Figure 9 shows the thirty campaign assets rendered for this paper, the eighteen of Figure 3 and twelve more, each as the input photograph with the box the model returned, the delivered asset at rest, and the same asset after a joint drive moved its listed joint. Table 4 gives the source input, the selected library entry, the support role, the applied scale, and the joint travel read back from the simulator. Twenty-seven inputs are single-object photographs and three are detections in cluttered images. The travel column also shows a limit of the source library: the driven joints of the printer and the kettle have millimeter ranges, so their two states coincide, and the bottle is a rigid entry with no joint. Figure 9: Thirty campaign assets. Each group: the input photograph with the grounding box, the delivered asset at rest, and the same asset with its driven joint moved, rendered from one camera in Isaac Sim; the link the drive moves is tinted in both states. Rows 1–6 are the assets of Figure 3. Table 4 lists inputs, entries, scales, and joint travel; the bottle is rigid, so its two states coincide. Table 4: Assets of the gallery figure. Rows follow the figure. Input: Open Images id prefix (OI) or the COCO image and detection index of a cluttered-image input. Entry: the PartNet-Mobility id or the Objaverse id prefix of the selected library entry. Role: free-standing or mounted support. Scale: the uniform scale the adapter applied. Joint and travel: type of the driven joint and its displacement between the two rendered states, read back from the simulator and signed along the joint axis; the bottle is a rigid entry with no joint. Asset Category Input Entry Role Scale Joint Travel Door Door OI 8d64efb8 9288 mounted 1.05 revolute 70∘ Window Window OI 5476ddec 103312 mounted 0.78 prismatic -36 cm Cabinet Storage OI 01c07a25 46277 free 0.91 revolute 85∘ Drawers Storage OI 26234158 46130 free 0.74 prismatic 26 cm Fridge Refrigerator OI a10ea74b 10849 free 0.47 revolute 100∘ Microwave Microwave OI 26315460 7310 free 0.38 revolute 85∘ Dishwasher Dishwasher OI 7ed08307 12592 free 0.54 revolute 63∘ Oven Oven OI 1f2dc2f1 7290 free 0.42 revolute 75∘ Washer Washer OI daded832 103361 free 0.57 revolute 90∘ Laptop Laptop OI 40365619 11586 free 0.39 revolute -85∘ Trash can Trash can COCO 1993, det. 7 11124 free 0.16 revolute 95∘ Toilet Toilet OI d60e1c25 102634 mounted 0.45 revolute 80∘ Trunk Suitcase OI 23455e0b 47645 free 0.44 revolute 144∘ Box Box OI c2ef3993 100129 free 0.15 revolute -98∘ Table Table OI d381c0b5 20453 free 0.79 prismatic 36 cm Fridge-freezer Refrigerator OI 9e21c789 10489 free 0.88 revolute 144∘ Base cabinet Storage OI 1b05c660 48855 free 0.69 prismatic 38 cm Lidded bin Trash can OI 2fd7ad92 101377 free 0.70 revolute 96∘ Display Display OI b432fc0f 4542 free 0.31 revolute -17∘ Printer Printer OI c61fbb3b 103872 free 0.33 prismatic 0.5 m Cabinet Storage OI c780bf71 47632 free 0.79 revolute 96∘ Lamp Lamp OI e916d0b 15913 free 0.90 revolute 36∘ Toaster Toaster COCO 159791, det. 2 103556 free 0.29 prismatic 9 cm Camera Camera OI 35e49214 102852 free 0.30 revolute 108∘ Coffee machine Coffee machine OI 460475ca 103092 free 0.38 revolute 108∘ Cabinet (1 door) Storage OI aea23ba3 45623 free 0.50 revolute 72∘ Microwave Microwave COCO 159791, det. 3 7366 free 0.28 revolute 72∘ Kettle Kettle OI 1784917c 102714 free 0.15 prismatic 1.2 m Table Table OI 3030fcc2 22241 free 0.99 prismatic 55 cm Bottle Bottle OI 17135174 8f67849a free 0.00035 rigid – A.5 Development configurations under the target-assisted protocol Before the formal campaign, successive versions of the pipeline ran on the same 2,000 inputs under the target-assisted protocol, which supplies the registered target region and category label with the image (Table 5). Its first three configurations vary scale handling with a single retrieved candidate: adapting the highest-ranked candidate passes 685, a calibrated scale estimate 1,043, and three scale hypotheses committed before any simulator feedback 1,172. Two further configurations add multimodal selection over a pool of five, so 1,172 is not their baseline. Selection alone passes 887; the derived track reuses that run and adds the articulation adapter, whose structure-preserving repair recovers 316 of the 558 cases with an invalid articulation, for 1,203. Among the cases that still do not pass, 311 are repair-uncertain, 230 insufficient-evidence, 242 exported-but-not-ready, 12 gravity failures, and 2 collision failures. One number needs separating out: the three-hypothesis result uses the hypothesis committed before replay, and crediting a case whenever any of the three would have passed gives 1,186/2,000, which depends on the outcome it is meant to score. Scale handling helps almost only rigid objects: across the three single-candidate configurations rigid inputs pass 27, 385, and 514 of 1,244, while the articulated count holds at 658/756. Library depth also bounds selection, since 337/2,000 five-candidate queries offer fewer than five distinct candidates, and the single- and five-candidate configurations draw on only 569 and 558 different library entries, a small part of the catalog. The formal campaign of Section 4.2 runs the final pipeline from scratch under the full-RGB protocol, so its 1,703 passes are not a row of this table. Table 5: Development configurations (target-assisted protocol). Each row received the registered target region and category label. All 2,000 inputs are retained. Rows are successive development versions, not one-factor ablations; the last row reuses the retrieval above it and adds the articulation adapter. Configuration Pass / 2,000 Rate (%) Single-candidate construction Single-candidate + adaptation 685685 34.2534.25 + scale estimation 1,0431,043 52.1552.15 + selected scale hypotheses 1,1721,172 58.6058.60 Candidate selection and repair Multimodal candidate selection 887887 44.3544.35 + articulation adapter 1,2031,203 60.1560.15 Table 6 reports three development configurations on the 237 annotated instances of the cluttered images, each given the registered target crop. Category-conditioned retrieval, whose pool is restricted to the registered label, exports all 237, passes 236, and carries 49/50 images to a full pass. Multimodal selection also sees that label but demands operated-part evidence in the crop, so it abstains 44 times, exports 192, passes 191, and completes 19/50 images. The crop-only control ranks from the crop with no label, and most of its candidates do not match the target; it exports 34, passes 29, and completes no image. These rows measure construction from an annotated crop; the detection-based result of Section 4.3 answers a different question. Table 6: Cluttered images under the target-assisted protocol. Three development configurations on the 237 annotated instances, each given the registered target crop. Object counts use N=237N=237; image counts use N=50N=50. These rates measure construction from a located target and are not comparable with the detection-based result of Section 4.3. Configuration Exported Passed Rate (%) All-pass images Crop-only control 34 29 12.212.2 0 Category-conditioned 237 236 99.699.6 49 Multimodal selection 192 191 80.680.6 19 A.6 Cluttered-image uncertainty The 575 detections of Section 4.3 are the objects the pipeline itself proposed on the 50 images, 289 on the 32 images of the first detection pass and 286 on the 18 re-detected ones (Appendix A.2); 563 of them pass the physical gate. A detection exists only where the pipeline localized something, so that share is a construction rate after localization and does not compare with the 85.15% of Section 4.2, which counts every input: set aside the 230 inputs that are not localized and the single-object benchmark passes 1,703 of 1,770. Under the full-RGB protocol of Section 4.3, the 172 localized and 162 passing instances of 237 carry Wilson intervals of 66.57–77.86% and 62.18–73.94%, and the 15/50 all-pass images 19.10–43.75%. Under the target-assisted protocol, object-level physical passes are 29/237 for the crop-only control, 236/237 for the category-conditioned configuration, and 191/237 with multimodal selection, with Wilson intervals of 8.66–17.02%, 97.65–99.93%, and 75.08–85.12%. At the image level, 0/50, 49/50, and 19/50 images have every annotated target clear its own object gate, a criterion that aggregates object physics and leaves semantics and the spatial relations between objects unverified. Because objects within one image are correlated, image-level intervals resample source images with 10,000 bootstrap replicates under seed 20260915. The grouped per-object records accompany the analysis. A.7 Library and timing diagnostics The library is one merged catalog of 11,372 entries, assembled once from 11,398 source objects; 26 were dropped because no previewable mesh could be built from their files. Objaverse (Deitke et al., 2023b) supplies 8,986 of the entries, restricted to objects released under C BY, PartNet-Mobility supplies 2,335 (Xiang et al., 2020), Google Scanned Objects 41, and the YCB object set 10. The articulation comes from the PartNet-Mobility entries: the catalog records revolute motion for 1,790 of them and prismatic motion for 1,040, an entry may have both, and the other 9,037 entries are rigid. The entries carry 283 category labels, 232 of them with ten entries or more, so the 31 benchmark categories touch only a part of it. Table 7: Library growth and retrieval diagnostics. Every setting answers the same 2,000 queries. Coverage: fraction whose category the library holds. Top-1: fraction whose selected asset carries the requested label. Latency covers retrieval only, excluding adaptation and validation. Library Entries Coverage (%) Top-1 (%) Median (ms) P95 (ms) Base 141141 46.1546.15 17.5017.50 44.8944.89 68.3868.38 More instances 418418 46.1546.15 22.3022.30 44.9644.96 68.4768.47 More mechanisms 472472 46.1546.15 24.7524.75 44.9444.94 68.5268.52 More categories 614614 100.00100.00 30.3030.30 45.0045.00 68.4168.41 Full 11,37211,372 100.00100.00 51.4551.45 50.1350.13 74.4374.43 Category coverage asks whether the requested category label occurs in the candidate pool; top-1 agreement asks whether the selected candidate carries that label. Both operate on category labels alone and verify neither the operated part nor the articulation of the photographed object. The denominator is 2,000 queries in every setting, and we measure what a library addition makes available separately from what the selector does with it. The recorded query times cover feature extraction, lookup, and ranking. They time retrieval alone and stop before adaptation and validation. The scale and multi-scale runtime fields hold identical values in all 1,963 available pairs, so they do not constitute two independent measurements. The clean timing comparison of Tables 1 and 9 runs every local method on cards of the same model (RTX 5090, 32 GB), one worker per card, used exclusively. It covers a 32-input sample of the registered subset and keeps the code, weights, and settings of the registered runs. One stage does not fit: the PhysX-Omni geometry stage needs up to 91 GiB over the 2,000 inputs, so it runs on an exclusive RTX PRO 6000 (96 GB). The physical gates run in their own lanes with an empty geometry cache. Runs that shared a card stay in the record and are excluded from the clean comparison. Loading, inference, adaptation, validation, wall time, and the cost of failed attempts are logged as separate fields. A.8 Controlled external-method and module studies PhysX-Anything, PhysX-Omni, PAct, TRELLIS.2, and PartCrafter run on the full 2,000 RGB inputs; Articulate-Anything and the GPT agent share a 200-input subset. The subset is drawn under seed 20260916, taking at least one input from each of the 31 source categories and allocating the remaining quota in proportion to category size. Sampling uses only the inputs; output quality plays no part in it. A source object whose motion type the photograph leaves ambiguous keeps its source label, and does not inherit the label of a candidate we selected. We designed the study with our earlier results already in hand, so its protocol is fixed in advance but not blind. Table 8 lists the subset by category. Table 8: Composition of the 200-input subset. Per category: inputs and physical passes of the full 2,000-input campaign and of the registered 200-input subset shared with the external methods (seed 20260916, at least one input per category, the rest in proportion to category size). The subset passes are a recount of the campaign records on those inputs, not a new run. Category All 2,000 Subset Category All 2,000 Subset In. Pass In. Pass In. Pass In. Pass Table 264 231 24 24 Toilet 25 20 3 2 Window 263 157 23 11 Keyboard 24 17 3 2 Bottle 159 127 15 10 Kettle 23 23 3 3 Chair 153 111 14 10 Coffee maker 19 19 3 3 Door 145 120 13 9 Printer 19 19 3 3 Storage 140 137 13 13 Washer 19 17 3 2 Camera 132 126 12 12 Oven 17 16 2 2 Phone 96 86 9 9 Trash can 17 17 2 2 Cart 89 87 9 8 Fridge 16 15 2 2 Box 61 57 6 5 Suitcase 16 15 2 2 Mouse 59 59 6 6 Clock 15 10 2 1 Knife 57 56 6 6 Microwave 5 5 1 1 Laptop 47 42 5 4 Remote 5 5 1 1 Display 45 42 5 5 Dishwasher 1 1 1 1 Lamp 35 33 4 3 Scissors 1 1 1 1 Pen 33 32 4 4 Total 2,000 1,703 200 167 The agent baseline runs gpt-6-astra with max reasoning, 32 tool rounds, a cumulative 128K output-token budget, and three hours per case. Each case is isolated and receives only its own input, the task, the modeling tools, and the same catalog access we use; our selected outputs and the evaluation answers are withheld. The configuration applies the agent workflow of Guo (2026), built for recorded multi-view robot episodes, to single photographs. The API relay ran out of credit after 18 of 200 cases. A refused call is an infrastructure outcome and not a model failure, so once the relay was topped up we reran the 182 refused cases with the same harness and model on another node. All 200 cases follow one protocol. The agent exports an asset for 199 of 200 inputs; 69 pass the native gate and 44 the adapted one. Articulate-Anything (Le et al., 2025) retrieves a PartNet-Mobility mesh and drives a commercial vision-language model through an API to write the articulation as code. Its first attempt stopped at the same relay balance after 26 of 200 cases and exposed two harness faults: 129 library objects lacked a readable reference render, and gemini-2.5-flash, which replaces the official default gemini-1.5-flash (not served by the relay), sometimes answers outside the JSON format of the official prompts. We completed the renders with the official preprocessing call (93 of the 129 objects render; the rest stay excluded, as in the official code) and added a normalizer that only reformats such replies for the unchanged official parsers, then reran all 200 cases. The gate reads the geometry-free structural links of its URDF as empty bodies, as it does for PartNet-Mobility assets. It exports an asset for 162 of 200 inputs; none declares mass or inertia, so only the adapted condition applies, and 70 pass it. PartCrafter (Lin et al., 2025) generates one mesh per part and TRELLIS.2 (Xiang et al., 2026) one textured mesh per photograph, both with their official code and checkpoints; PartCrafter takes its part count from a vision-language model through the same API relay. Neither delivers mass, inertia, or joints, so the native condition does not apply to them and their meshes enter only the adapted condition as a single rigid body. The released PAct dataset loader takes a part-label mask and reads the number of parts from it. Under the full-RGB protocol we therefore supply an automatic part-mask producer (Grounding DINO (Liu et al., 2024) with SAM (Kirillov et al., 2023); Table 1). A fully opaque mask, or a rectangle taken from our annotations, would hand the method its part count for free, so neither works as a stand-in. Figures 10 and 11 show what every method delivers for twelve subset inputs. Figure 10: Qualitative comparison of all methods. Columns: the input (cropped around our grounding box for display; every method saw the whole image), AffordCraft, PhysX-Anything, PhysX-Omni, PAct, PartCrafter, TRELLIS.2, Articulate-Anything, and the GPT-6 Astra agent; rows: six registered-subset inputs of articulated categories (refrigerator, microwave, laptop, storage furniture, door, washing machine). Each asset is shown as delivered, at rest, from one camera in Isaac Sim with a uniform clay material; links that a movable joint drives are tinted (for AffordCraft, the operated link). PhysX-Anything, PhysX-Omni, PAct, and the agent appear as our gate reads their native exports, and Articulate-Anything’s URDF in the same layout; PartCrafter and TRELLIS.2 deliver meshes only, and Articulate-Anything exports nothing for row 4. Every AffordCraft asset passes the physical gate. Of the external outputs, PhysX-Anything passes in row 5 as delivered and in row 4 after our adaptation, TRELLIS.2 in rows 3 and 4 after adaptation, Articulate-Anything in rows 2, 3, and 5 after adaptation, and the agent in rows 1, 2, and 6 as delivered and in rows 2 and 3 after adaptation; every other output fails both conditions. Figure 11: Additional qualitative comparison. Columns as in Figure 10. Rows: for each of six further articulated categories (oven, toilet, box, kettle, window, suitcase), the first registered-subset input, in the order of the subset, for which every method delivered an asset. Every AffordCraft asset passes the physical gate. Of the external outputs, PhysX-Anything passes in rows 5 and 6 as delivered, PhysX-Omni in rows 1, 4, and 6 as delivered and in rows 4 and 6 after our adaptation, PAct in row 1 as delivered, Articulate-Anything in rows 1, 3, 4, and 5 after adaptation, and the agent in rows 1, 3, and 4 as delivered; PartCrafter and TRELLIS.2 fail every row, and no external output passes for the toilet. Each external method is scored on its native output and again after our common physical adaptation, as two separate conditions, and we log the source revision, checkpoint, and any compatibility change behind every run. Our common adaptation builds collision geometry for every delivered part; an input whose build the memory guard or the build time limit stopped at its first gate and again in a re-gate with more memory per build counts as an adapted failure (PAct 69 of 2,000 inputs, PartCrafter 10, PhysX-Omni 6). The run records keep an infrastructure block, an unfinished run, a geometry-only output, a physical failure, and a completed task apart, and Table 1 reports completed runs only. The AffordCraft row of that table is the formal campaign itself, which already runs under this protocol; its 167/200 on the registered subset recounts the campaign’s verified per-case records and involves no new run. The module study varies one factor at a time: the encoder, task conditioning, multimodal selection, scale adaptation, physical-feedback reselection, deterministic repair, installation evidence, the decomposition cascade, or the articulation adapter (Table 2). Without task conditioning, retrieval is not restricted to the named category and the selector does not see the instruction; without the selector, retrieval order decides; without reselection, the first constructed candidate is final; without installation evidence, every asset is free-standing. The decomposition cascade tries five CoACD (Wei et al., 2022) settings of increasing resolution until the collision geometry passes its audit, and the ablated condition keeps only the first. Selector and adapter comparisons hold the candidate pool fixed, and an encoder change rebuilds the index while leaving the catalog and every downstream setting alone. Every row is the complete run of its condition; no row takes the best of several related runs. A.9 Resource-reporting fields The columns of Table 1 are defined as follows. Export is the share of inputs that end with a structured asset. Native scores that asset as delivered at the physical gate of Section 3.3, adapted scores it after our common physical adaptation, and articulated is the share of inputs whose accepted asset carries at least one movable joint. Median time is the median end-to-end (E2E) wall-clock time from input to validated asset in the clean timing run. GPU minutes per pass divides that time by the pass rate of the row, giving the cost of an accepted asset at the median; a worker holds one GPU, so its worker-minutes are GPU-minutes. Pass in 5 min is the share of the timed inputs whose asset clears the gate within five minutes of wall time, one point of the curves in Figure 5, which scores each external method on its native path where that applies and on the adapted one otherwise, with the time of that build. Peak is the largest device memory in use on the worker’s GPU. For every local method these timing and resource cells come from the clean timing run (one worker per exclusive RTX 5090, an RTX PRO 6000 for the PhysX-Omni geometry stage; 32 inputs of the registered subset) and the pass rates from the registered runs. Table 9 lists the stage timing, wall-clock, throughput, GPU, and monetary measurements of the matched full-RGB comparison for every method. The E2E median covers the complete construction and physical-validation path, not just retrieval. Throughput counts physically accepted assets per hour; total cost per valid asset includes failed attempts. API expense applies the model providers’ official list prices to the token counts each run recorded: gpt-6-astra at OpenAI’s Standard rates (10 USD per million input tokens, 1 for cached input, 12.50 for cache writes, 50 for output), gemini-2.5-flash and gemini-3.8-flash at Google’s (0.30/2.50 and 0.75/3.75 USD). The relay reports cached but not written prompt tokens, so every uncached agent prompt token is billed as a cache write, an upper bound (2.56 USD per input at the plain input rate). PartCrafter’s run keeps each part-count reply but not its token count; its 0.005 USD per input uses the tokens the relay billed for 569 of its calls. The two API routes were not part of the clean timing run (the relay had no credit then); their times are those of their own runs over the 200 subset inputs. AffordCraft is timed with the geometry of its library entries built once, the regime in which every later query reuses it: a median of 40.0 s per input. Run on the same 32 inputs from the geometry cache as it stood before the formal campaign, it takes a median of 120.8 s, or 2.4 instead of 0.8 worker-minutes per accepted asset at the median; the difference is geometry built once and reused afterwards. Within five minutes of wall time, 53.1% of the timed inputs pass in this regime and 43.8% from the pre-campaign cache, against at most 34.4% for any other route (Figure 5). Twelve of the 32 inputs are accepted within 30 s, before any other method has delivered one. PhysX-Anything, the strongest generative method, accepts its first input in the first minute and levels off at 20 of 32 after 15 minutes, where AffordCraft stands at 21 and ends at 25. The two API routes accept 16% (Articulate-Anything) and 6% (the agent) of their 200 inputs within five minutes, and level off at 35% and 34.5%. Means are higher (711 and 1,340 s) because a few inputs work through many candidates and failed constructions are not cached (the slowest takes 8,825 s). Table 9: Matched resource accounting on the full-RGB comparison. All times come from the clean timing run (one worker per exclusive RTX 5090; the PhysX-Omni geometry stage on an RTX PRO 6000) over a 32-input sample of the registered subset, AffordCraft with the geometry of its library entries built once; pass rates from the registered inputs of each Table 1 row. Per-input medians of the stage wall times each stage process recorded; adaptation is our common physical adaptation of the delivered asset (not applicable to AffordCraft, whose generation row is its own adaptation); E2E, minutes per accepted asset, throughput and GPU time follow the native path where it applies and the adapted path (‡ ) otherwise. API cost: official list prices of the model providers applied to the token counts each run recorded (gpt-6-astra: OpenAI Standard input, cached-input and cache-write rates; Gemini models: Google), 0 for methods that run only local models; § @sectionsign: times of the API run itself. Metric Afford- Craft PhysX- Anything PhysX- Omni PAct Part- Crafter TRELLIS.2 Articulate- Anything GPT agent Perception / retrieval (s) 15.6 199.0 288.3 2.0 22.2 0.7 115.2 Generation (s) 37.9 104.3 125.0 94.0 46.0 142.7 58.5 1414.3 Adaptation to our contract (s) 237.2 287.8 332.2 300.4 253.1 168.8 64.7 Physical validation (s) 2.5 2.7 2.4 4.3 13.2 E2E median (s) 40.0 303.3 464.6 108.8 230.0‡ 393.9‡ 349.6‡§ 1423.0§ Throughput (passes / worker-h) 75.0 5.3 2.8 7.8 1.0‡ 2.2‡ 3.6‡§ 0.9§ Minutes per accepted asset (median) 0.8 11.3 21.2 7.7 62.8‡ 27.8‡ 16.6‡§ 68.7§ Peak GPU memory (GiB) 21.2 19.6 60.5 21.8 13.6 21.5 API cost (USD / input) 0 0 0 0 0.005 0 0.068 2.90 For reference, Table 10 lists the wall-clock times the formal campaign itself recorded over the 2,000 single-object inputs. The campaign was a throughput run, with concurrent workers per GPU and a geometry cache that the same frozen builder had computed in advance, so these are not exclusive-device latencies; the clean timing run supplies the timing cells of Tables 1 and 9 instead. Per pass, the campaign spent 11.8 worker-minutes over all 2,000 inputs and 12.8 over the 200-input subset, failed inputs included. The median input completes in 85.0 s, and per input the median perception, adaptation, and validation times are 31.6, 67.3, and 6.5 s. The mean of 602.4 s and the P90 of 1,629 s reflect a long tail: an input may work through up to 20 retrieved candidates in shortlists of five, the mean input considered 2.9, and each construction attempt has a 600 s materialization budget. Each worker held one A100 for its whole run. Its process tree, which holds the multimodal model and two physics processes, peaked at 29.0 GiB of GPU memory at the median worker and 32.3 GiB at most, and loading the model took 16.2 s at the median. The visual index of the full library was built once, offline, in 1,190 s. Table 10: Recorded timing of the formal campaign. Wall-clock seconds over the 2,000 single-object inputs, from the verified per-input records of a throughput run (concurrent workers per GPU, pre-computed geometry cache), so not exclusive-device latencies. Phase rows: median and mean over the n recorded executions of the phase; repair runs at most once per attempt, after a failed export or a failed construction check, and returned a repaired asset in 179 attempts. Group rows: per input, over the n inputs that reached the group. End to end: per input; the P90 is 1,629 s. Stage Group n Median (s) Mean (s) Grounding perception 2,000 5.78 5.95 Retrieval perception 1,767 0.09 0.10 Multimodal ranking perception 2,347 25.00 25.64 Asset construction adaptation 4,781 93.61 225.73 Repair adaptation 3,138 0.00 10.03 Construction physics validation 2,316 2.91 5.42 Final physics validation 1,703 2.81 5.47 Perception, per input 2,000 31.57 36.12 Adaptation, per input 1,767 67.30 628.56 Validation, per input 1,712 6.52 12.77 End to end, per input 2,000 85.00 602.39 A.10 Single-asset manipulation tasks Besides the ten composed scenes of Appendix A.11, we train on ten single-asset tasks, one per constructed asset (Table 11, Figure 12), so the policy study covers twenty tasks. Five rigid objects (bottle, box, camera, clock, display) must be pushed fully into a marked goal region and left stable; five articulated assets (a microwave, three storage units, and a drawer unit) must have their door or drawer closed into the registered closed range and held there. Data and policy. A scripted teacher with privileged state provides 100 successful episodes per task, 1,000 in total with 245,954 frames, frozen before training together with 200 held-out initial states, 20 per task. The robot, the rates, and the action format match the composed scenes: a Franka Panda commanded at 10 Hz over 60 Hz physics in Isaac Sim, with seven arm-joint targets and two finger widths predicted as eight-step chunks of which four execute. The only learned component is a recurrent head with proprioceptive feedback over frozen OpenVLA-OFT features from the public LIBERO-Spatial checkpoint (Kim et al., 2025; Liu et al., 2023a); it observes one camera image, the nine joint positions, and the task instruction. The teacher’s actions, the task stage, and the object state are withheld from it. It trains for 32,000 steps on the teacher episodes only, with correlated 0.005 rad noise added to the arm joint inputs of 75% of the sequences. We then evaluate the checkpoint the run registered at the end, fixed before any evaluation, and not a step picked once the numbers were in. Evaluation and results. The trained head and the untrained head (the same head with its random initial weights) run the same 200 held-out initial states, and success is read from simulator state and contacts, not from video. The trained head completes 114/200 episodes (57.0%, Wilson 95% interval 50.1–63.7%) against 1/200 for the untrained head, 58/100 on the rigid and 56/100 on the articulated tasks, and it completes every task in at least 3 of its 20 episodes. Five initial states of the articulated tasks produced no episode record in either condition and count as failures; without them the rates are 114/195 and 1/195. Of the trained head’s failures, 34 of 42 on the rigid tasks leave the object outside the goal region or unstable, and 24 of 39 on the articulated tasks command joint targets outside the admissible range. These ten assets were also in use while the suite was developed. The study therefore shows that such tasks can be learned from constructed assets; it says nothing about unseen ones. Table 11: Ten single-asset tasks. Successes out of 20 held-out episodes per task for the same head before and after training; blocked: initial states without an episode record in either condition, counted as failures. Asset Task Untrained Trained Blocked Rigid objects Bottle push into region 0 5 – Box push into region 0 6 – Camera push into region 0 19 – Clock push into region 0 14 – Display push into region 0 14 – Articulated assets Microwave close door 1 8 2 Storage 02 (45526) close door 0 7 3 Storage 03 (45638) close door 0 3 – Storage 04 (46889) close door 0 18 – Drawer (46236) close drawer 0 20 – All ten tasks 1/200 114/200 5 Figure 12: Single-asset tasks. Start, midpoint, and final frames of one recorded held-out episode of the trained head per task; the mark gives that episode’s outcome, and the numbers give untrained → trained successes out of 20 held-out episodes. Crops stay fixed within each sequence. A.11 Composed-scene manipulation study Scenes. Table 12 lists the ten scenes. The articulated assets are library entries the pipeline selected and adapted for campaign inputs, drawn from PartNet-Mobility (Xiang et al., 2020); the cup and box are Objaverse entries (Deitke et al., 2023b) that passed the same gate. All are baked at scene scales of 0.35–1.0 and anchored to the table by a fixed joint, and the microwave stands on a 10 cm platform. Each episode draws from its seed the furniture position (1.5–2 cm jitter), the task object’s position inside a scene-specific region and yaw (±8∘± 8 ), the distractor’s identity and pose from a pool of seven rigid entries, and the camera pose (±2∘± 2 azimuth, ±1∘± 1 elevation). In the four door scenes the door starts ajar at 20–30∘; the drawer and the trash-can lid start closed and the laptop open; the two transfer scenes start with the drawer pulled out 11 cm, the cabinet or refrigerator door open at 85–90∘, and the cup inside the drawer. The robot is a Franka Panda with seven arm joints and two fingers, commanded at 10 Hz over 60 Hz physics in Isaac Sim 5.1 (NVIDIA, 2025), with arm commands clamped to 0.06 rad per tick. Table 12: Per-scene tasks, assets, and results. Asset ids are PartNet-Mobility entries for articulated objects and 8-character Objaverse id prefixes for rigid ones. Teacher counts are out of 32 episodes. Trained-head columns show task successes and first-sub-goal counts on held-out states (10 episodes) and task successes on seen states (6); sg1: first sub-goal reached. Eight scenes include one distractor object. Scene and task Assets Teach. /32 Held-out task ⋅· sg1 /10 Seen /6 Untr. /4 Cabinet (2-door) + cup: open far door, put cup inside 46277, fae08c03 20 0⋅·10 0 0 Cabinet (1-door) + box: open door, put box inside 45623, 160d40d9 24 4⋅·10 3 0 Refrigerator + cup: open door, put cup inside 10849, fae08c03 32 0⋅·10 0 0 Microwave + box: open door, put box inside 7310, 160d40d9 28 1⋅·9 2 0 Drawer + cup: open top drawer, put cup inside 46130, fae08c03 29 3⋅·8 2 0 Drawer + two cups: open top drawer, put both inside 46130, fae08c03×2 29 0⋅·8 0 0 Drawer→ + cup: move cup from open drawer to open cabinet, close drawer 46130, 46277, fae08c03 32 2⋅·2 1 0 Drawer→ + cup: move cup from open drawer to open fridge, close drawer 46130, 10849, fae08c03 31 1⋅·1 0 0 Laptop + cup: close lid, put cup on top 11586, fae08c03 27 0⋅·7 0 0 Trash can + box: lift lid, drop box inside 11124, 160d40d9 21 0⋅·1 0 0 Total 273/320 11/100 ⋅· 66/100 8/60 0/40 Table 13: Training configurations. Each row is one trained head. Episodes count teacher demonstrations plus mixed rollouts after a dynamics filter. The first row used 4 held-out episodes per scene; the others use 10 held-out and 6 seen. Sg1: first sub-goal reached. Untrained heads complete 0 held-out episodes in every configuration. Training set Held-out Seen Config. Teach. Roll. Frames Task Sg1 Task Sg1 3rd-person, teacher only 272 0 152,469152,469 6/40 28/40 8/40 29/40 + wrist, crop; round 1 267 82 205,129205,129 9/100 67/100 8/60 43/60 Round 2 (all rollouts) 267 397 373,031373,031 11/100 66/100 8/60 42/60 Teacher and evaluator. The teacher is a scripted planner with access to simulator state that grasps door edges, drawer rims, and the lid overhang with closed-loop arcs and picks objects at their rim. It completes 273/320 episodes; its 47 failures split into 22 before and 25 after the first sub-goal. The evaluator reads only the recorded per-tick trace and the scene goals; the teacher’s plan is not available to it. An articulation goal requires the joint to pass a threshold: doors and lid at least 60–70∘, drawer at least 7 cm, laptop lid at most −75∘-75 . A placement goal requires the object’s center inside the target link’s bounding box, with the gripper open and out of contact, moving at most 1 cm per tick over the final ten ticks. The transfer scenes ask for one thing more, the drawer pushed to within 3 cm of closed with the cup still inside. An object that falls below the table fails the episode. Success is read from state, never judged from video, and every episode keeps its video and trace. Table 14: Instructions and data-collection rounds per scene. Under each scene stands the instruction string every teacher and policy episode of that scene received. Each round lists task successes ⋅· first-sub-goal completions over its episodes: the teacher round (32 per scene) and the two rounds of mixed rollouts driven by the first and second trained heads (20 and 24 per scene). Ticks: median length of the passed teacher episodes at 10 Hz. Train: episodes of the scene in the final training set after the exclusions described under Data and training (Table 13, last row). Scene and instruction Teacher task ⋅· sg1 /32 Rollouts 1 task ⋅· sg1 /20 Rollouts 2 task ⋅· sg1 /24 Ticks Train Cabinet (2-door) + cup 20⋅·24 8⋅·20 10⋅·24 528 54 open the cabinet door fully and put the cup inside the cabinet Cabinet (1-door) + box 24⋅·31 14⋅·19 18⋅·24 573 67 open the cabinet door fully and put the box inside the cabinet Refrigerator + cup 32⋅·32 10⋅·20 10⋅·24 716 73 open the refrigerator door fully and put the cup inside the refrigerator Microwave + box 28⋅·32 12⋅·20 12⋅·24 569 67 open the microwave door fully and put the box inside the microwave Drawer + cup 29⋅·32 10⋅·17 11⋅·20 385 66 open the top drawer and put the cup inside the drawer Drawer + two cups 29⋅·32 6⋅·19 6⋅·20 640 68 open the top drawer and put both cups inside the drawer Drawer→ + cup 32⋅·32 5⋅·5 9⋅·9 480 69 take the cup out of the drawer, put it into the cabinet and close the drawer Drawer→ + cup 31⋅·31 1⋅·1 6⋅·6 451 64 take the cup out of the drawer, put it into the refrigerator and close the drawer Laptop + cup 27⋅·30 10⋅·16 11⋅·19 546 71 close the laptop and put the cup on top of the laptop Trash can + box 21⋅·22 11⋅·11 10⋅·14 770 65 open the trash can lid and put the box into the trash can Total 273⋅·298 /320 87⋅·148 /200 103⋅·184 /240 664 Policy. The policy observes a fixed third-person camera (1280×960 render, cropped to the workspace and resized to 320×240), a wrist camera on the hand (70∘ field of view, 320×240), and its nine joint positions. Both images pass with the instruction through the frozen OpenVLA-OFT backbone (Kim et al., 2025), using its public LIBERO-Spatial checkpoint (Liu et al., 2023a). The mean of its 56 action-token states gives a 4,096-d feature per image. The two features, concatenated, feed a recurrent head (GRU, 512 units) that predicts eight future joint targets and executes four at 10 Hz. The backbone stays frozen, and the untrained condition is the same head with its random initial weights. Data and training. Four collection workers recorded eight teacher episodes per scene each (320 episodes, 273 successes). A first head trained on 143 successful episodes drove 200 mixed rollouts in the manner of DAgger (Ross et al., 2011): the policy executes bursts of 8–40 ticks inside the teacher’s plan, in half of the rollouts also its first up to 200 ticks, at a target executed share of 0.35, while the teacher supplies every label. A second head, trained on 267 teacher successes and the 82 successful rollouts, drove 240 more. The final head trains on the 267 teacher successes and 397 of the 440 rollouts, 214 of them unsuccessful, 373,031 frames in total; the 216 rollouts that did not both pass and complete the teacher’s plan are truncated 45 ticks before their end. Excluded are teacher failures, rollouts shorter than 165 ticks or with an object off the table, and 20 episodes whose executed eight-tick joint change exceeds 0.75 rad. Training runs 24,000 AdamW steps (batch 16, learning rate 3×10−43× 10^-4, cosine decay) on one A100 in 68 minutes, and again we evaluate the checkpoint registered at the end. Figure 13: Held-out episodes of the trained head, twelve frames each. The five scenes the head completed on held-out states (left column of Figure 7), each as its first passed held-out episode at twelve evenly spaced control ticks from the start to the final tick. Upper row: the third-person observation; lower row: the wrist camera at the same ticks. Both are the 320×240 images the policy received. Evaluation. Held-out episodes draw initial-state seeds that collection did not use, 10 per scene for the trained head and 4 for the untrained head; seen episodes replay the first six initial states of collection worker 0. Each episode has a budget of 1,000 control ticks. Table 13 lists the three trained configurations: adding the wrist view, the crop, and the rollouts changed neither first-sub-goal completion (28/40, 67/100, 66/100) nor task completion (6/40, 9/100, 11/100) beyond the overlap of their intervals. Seen and held-out states fail at similar rates, so the gap to the teacher is not a gap across initial states. The 55 failures after the first sub-goal divide into 30 that close the gripper beside the object without securing it, 12 that lift it and then release it outside the target volume, and 13 that never close on it. Of the 34 failures before the first sub-goal, 17 leave the door, drawer, or lid short of its threshold, 9 of them in the trash-can scene, where the lid stays untouched. The other 17 are transfer episodes that do not carry the cup out of the drawer: 13 fail the grasp, and 4 lift the cup without placing it. Results. On held-out initial states the trained head completes 11/100 episodes and reaches the first sub-goal in 66, against 0/40 and 1/40 for the untrained head (Table 15); on seen states it completes 8/60. Five scenes account for all of its completions, the left column of Figure 7: one-door cabinet and box 4/10, drawer and cup 3/10, drawer to cabinet 2/10, microwave and box 1/10, and drawer to refrigerator 1/10, with the first sub-goal reached in 10, 8, 2, 9, and 1 of 10 episodes. The other five yield no completion, although the head still reaches the first sub-goal in most of them: 10 of 10 for the two-door cabinet and for the refrigerator, 8 for the drawer with two cups, 7 for the laptop, and 1 for the trash can. These ten scenes also served during development, so the study does not measure generalization to unseen assets. Table 15: Manipulation in ten composed scenes. An independent evaluator reads success from simulator state. Held-out and seen refer to initial states unseen or seen during data collection; the teacher row scores the assets, not a policy. The untrained head uses random initial weights. Condition Episodes Task success Rate (%) Sub-goal 1 Scripted teacher 320 273 85.385.3 298 Untrained head, held-out 40 0 0.00.0 1 Trained head, held-out 100 11 11.011.0 66 Trained head, seen 60 8 13.313.3 42 Table 16: Outcome of the 100 held-out episodes of the trained head. Ten episodes per scene, classified from the recorded trace. After the first sub-goal: the gripper closed beside the object without securing it (grasp missed), lifted it and released it outside the target volume (lifted, not placed), or never closed on it (not grasped; one of the 13 approached without closing). Before the first sub-goal: the door, drawer, or lid stayed untouched or moved partly (mechanism columns), or, in the two transfer scenes, the cup was not carried out of the drawer (grasp missed, or lifted and dropped back). After first sub-goal Mechanism short Transfer, cup not out Scene Pass Grasp Lifted Not grasped Untouched Partial Grasp Lifted Cabinet (2-door) + cup – 8 1 1 – – – – Cabinet (1-door) + box 4 4 2 – – – – – Refrigerator + cup – 5 1 4 – – – – Microwave + box 1 3 5 – – 1 – – Drawer + cup 3 5 – – 1 1 – – Drawer + two cups – 5 3 – – 2 – – Drawer→ + cup 2 – – – – – 5 3 Drawer→ + cup 1 – – – – – 8 1 Laptop + cup – – – 7 – 3 – – Trash can + box – – – 1 9 – – – Total 11 30 12 13 10 7 13 4 Table 17: Action-head configuration of the scene study. Values of the registered round-2 training run. The backbone is never finetuned; the untrained condition uses the same head at initialization. Setting Value Backbone OpenVLA-OFT, LIBERO-Spatial checkpoint, frozen Per-image feature mean of the 56 action-token states, 4,096-d Images per step 2 (third-person crop, wrist), 320×240 Feature input to head 8,1928,192-d (two images concatenated) Proprioception 9 joint positions (7 arm, 2 finger) Head GRU, 512 units Action chunk 8 future targets, 4 executed, 9-d Arm targets delta to the chunk-start joint state, per-horizon scaled Control / physics rate 10 Hz / 60 Hz Optimizer AdamW, learning rate 3×10−43× 10^-4, weight decay 10−410^-4 Schedule 250 warm-up steps, cosine decay, 24,00024,000 steps Batch size 16 Motion loss weight 4.0 (scale 0.01 rad) Temporal consistency weight 0.25 Query overlap weight 0.1 Proprioception noise std 0.005 rad, correlation 0.8, p = 0.75 Seeds 138000001 (training), 138000002 (head init), 167031 (augmentation) Prompt In: What action should the robot take to instruction? : (instruction in lower case) Episodes, collection rounds, and configuration. Figure 7 shows four frames of each episode from the third-person camera: the start, the first sub-goal, the lift apex, and the final state (in the two transfer scenes: start, lift apex, release, and final state). Figures 13 and 14 show the ten episodes of Figure 7 at twelve evenly spaced ticks from both cameras the policy observes; the full third-person video of each of these ten episodes, one frame per control tick and unedited, is provided as supplementary material. Table 14 lists the instruction of each scene and the outcome of the three collection rounds, Table 16 classifies the 100 held-out episodes of the final head by scene, and Table 17 gives the head’s configuration. Figure 14: Scripted-teacher episodes, twelve frames each. The five scenes the trained head never completed on held-out states (right column of Figure 7), each as the first passed teacher episode of collection worker 0 at twelve evenly spaced ticks. Upper row: third-person observation; lower row: wrist camera. These episodes are demonstrations the head was trained on, not policy rollouts.