Paper deep dive

DeepInsight II: One Trace from Benchmark to Robot

Siyi Li, Yuchen Kang, Wuliang Wang, Zhengjie Zhang, Jiangpin Liu, Jianhao Yao, Jie Chen

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 68

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%

Last extracted: 8/23/2026, 2:21:45 AM

Summary

DeepInsight II extends the DeepInsight evaluation framework to bridge the gap between simulation and physical robot execution. It unifies evaluation across the Physical AI stack (System 2, 1, and 0) by maintaining a consistent trace identity from benchmark execution to real-robot trials. The paper demonstrates native-protocol fidelity for navigation and manipulation benchmarks (System 1) and introduces MotionBench to compare whole-body controllers (System 0) across simulation and hardware. It also proposes a repair-oriented diagnosis method using handoff labels to localize and fix failures in composed System 2-1-0 episodes.

Entities (15)

DeepInsight II · framework · 98%DeepInsight v1 · framework · 95%Habitat-Sim · simulator · 95%LIBERO · benchmark · 95%MuJoCo · simulator · 95%R2R-CE · benchmark · 95%RxR-CE · benchmark · 95%XPENG Robotics · organization · 95%MotionBench · benchmark · 92%JanusVLN · model · 90%RynnBrain-Nav · model · 90%StreamVLN · model · 90%System 0 · ai-layer · 90%System 1 · ai-layer · 90%System 2 · ai-layer · 90%

Relation Signals (11)

DeepInsight II → extends → DeepInsight v1

confidence 95% · DeepInsight II keeps that substrate fixed and quantifies the embodied half... The first DeepInsight report (v1) unified evaluation across this stack

DeepInsight II → producesby → XPENG Robotics

confidence 95% · XPENG ROBOTICS DeepInsight II: One Trace from Benchmark to Robot

RxR-CE → usessimulator → Habitat-Sim

confidence 95% · We evaluate R2R-CE and the English RxR-CE val-unseen split in Habitat-Sim

LIBERO → usessimulator → MuJoCo

confidence 95% · LIBERO MuJoCo/robosuite

R2R-CE → usessimulator → Habitat-Sim

confidence 95% · We evaluate R2R-CE ... in Habitat-Sim

DeepInsight II → includesbenchmark → MotionBench

confidence 92% · Second, MotionBench places four released whole-body controllers under one workload and metric contract

StreamVLN → evaluatedon → R2R-CE

confidence 90% · The evaluated cohort consists of the released StreamVLN... We evaluate R2R-CE

RynnBrain-Nav → evaluatedon → R2R-CE

confidence 90% · The evaluated cohort consists of the released ... RynnBrain-Nav ... We evaluate R2R-CE

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Across a Physical AI stack, evaluation maturity is inversely aligned with deployment risk: foundation models enjoy mature, standardized harnesses, while the embodied layers on which deployment actually turns remain fragmented across benchmark-specific simulators, embodiments, and interfaces. The first DeepInsight report (v1) unified evaluation across this stack behind three abstractions---task, resource, and result---but its quantitative evidence centered on the foundation-model layer; navigation and manipulation (System 1) and whole-body control (System 0) remained simulation case studies, and physical execution was outside its empirical scope. DeepInsight II keeps that substrate fixed and quantifies the embodied half. First, it reproduces released-checkpoint references across two navigation and four manipulation benchmarks under their native protocols. Second, MotionBench places four released whole-body controllers under one workload and metric contract, then carries a qualified within-family cohort from parallel simulation to matched real-robot trials in which simulated and physical rollouts share a parent trace identity while retaining execution-domain-specific records, making the sim-to-real gap a native reduction rather than a reconciliation across toolchains. Third, a composed System 2--1--0 study extends trace localization into five evidence-grounded handoff labels, each mapped to a concrete repair action, with a measured repairability criterion and physical episodes testing the same attribution under hardware-observable state. The contribution is therefore not a new evaluation architecture, but empirical continuity from benchmark execution to matched robot evidence and repair-oriented diagnosis.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

64,235 characters extracted from source content.

Expand or collapse full text

XPENG ROBOTICS DeepInsight I: One Trace from Benchmark to Robot Siyi Li, Yuchen Kang, Wuliang Wang, Zhengjie Zhang, Jiangpin Liu, Jianhao Yao, Jie Chen † XPENG Robotics Abstract Across a Physical AI stack, evaluation maturity is inversely aligned with deployment risk: foundation models enjoy mature, standardized harnesses, while the embodied layers on which deployment ac- tually turns remain fragmented across benchmark-specific simulators, embodiments, and interfaces. The first DeepInsight report (v1) unified evaluation across this stack behind three abstractions— task, resource, and result—but its quantitative evidence centered on the foundation-model layer; navigation and manipulation (System 1) and whole-body control (System 0) remained simulation case studies, and physical execution was outside its empirical scope. DeepInsight I keeps that substrate fixed and quantifies the embodied half. First, it reproduces released-checkpoint references across two navigation and four manipulation benchmarks under their native protocols. Second, MotionBench places four released whole-body controllers under one workload and metric contract, then carries a qualified within-family cohort from parallel simulation to matched real-robot trials in which simulated and physical rollouts share a parent trace identity while retaining execution- domain-specific records, making the sim-to-real gap a native reduction rather than a reconciliation across toolchains. Third, a composed System 2–1–0 study extends trace localization into five evidence-grounded handoff labels, each mapped to a concrete repair action, with a measured re- pairability criterion and physical episodes testing the same attribution under hardware-observable state. The contribution is therefore not a new evaluation architecture, but empirical continuity from benchmark execution to matched robot evidence and repair-oriented diagnosis. 1 Introduction A policy headed for hardware is evaluated most rigorously where the stakes are lowest. Its semantic reasoning (System 2)—the layer farthest from physical execution, increasingly packaged as a dedicated robot brain [1]—runs through mature harnesses: standardized datasets, shared inference backends, published protocols. Its navigation and manipulation skills (System 1) are scored by whichever benchmark each was trained against, each binding its own simulator, embodiment, and policy interface. And its whole-body controller (System 0)—the layer whose failures are physical—is evaluated through per-lab harnesses whose readings rarely transfer. The layers on which deployment actually turns are the ones evaluated with the least shared infrastructure. Figure 1 previews the response this report develops. The difficulty is structural rather than a matter of packaging. Embodied episodes are closed-loop: every action changes the observation presented at the next step, so evaluation is a stateful rollout against a live simulator or robot rather than a batched pass over a static dataset [2–4]. The resulting heterogeneity has three coupled forms: workloads differ in task and metric semantics; systems under test differ in architecture, checkpoint, controlled DoF, and native input–output interface; and execution substrates range from parallel simulators to serial, safety- constrained physical robots. Changing any one of these typically changes the harness and the record against which the result is interpreted: readings produced by separate harnesses do not automatically share episode identity, execution semantics, or reduction rules, and aggregate success hides the local failures and simulation–reality discrepancies that govern deployment. Recent infrastructure efforts isolate dependencies and federate benchmark servers [5], but typically leave each benchmark’s episode loop and result representation intact; real-robot evaluation is then joined post hoc, if at all. The objective, therefore, is not to erase this fragmentation with one universal benchmark or embodiment, but to make the heterogeneity explicit while preserving comparable evaluation semantics. The first DeepInsight report [6] (v1) established the substrate for this purpose: onereset/stepepisode driver, oneacquire/releaseresource protocol, and one identity-addressed trace across the System 2–1–0 stack. Its quantitative evidence, however, centered on System 2; the embodied layers remained simulation-only case studies, † Correspondence: Jie Chen at chenj81@xiaopeng.com 1 arXiv:2608.16556v1 [cs.AI] 17 Aug 2026 DEEPINSIGHT I embodiment = simembodiment = real · Unitree G1 system 1 released checkpoints run under native protocols · every episode leaves real frames R2R-CE · Habitat-Sim egocentric frametop-down trajectory LIBERO · robosuite resetapproachcontactsuccess / failure system 0 one MotionBench motion · matched at the same t₀–t₃ sim real t₀t₁t₂t₃ full system a composed System 2–1–0 episode navigate_to_targetH2greet_user→ ... → terminate navigatefailure intervaldiagnosed sim → realmatched H2 evidence · simulation and physical robot sim · before repairsim · after repairreal · before repairreal · after repair HCV repair traceone identity-addressed run_id per rollout — benchmark rollout · MotionBench parent trace · composed episode simulation branchphysical branch DeepInsight I does not merely run embodied benchmarks; it preserves episode identity from benchmark execution to physical evidence and repair-oriented diagnosis. Figure 1. The embodied evidence of this report, on one runtime: System 1 checkpoints under native benchmark protocols, a System 0 motion matched between simulation and the physical robot, and a composed System 2–1–0 episode diagnosed down to a concrete repair and re-tested in simulation and on the physical robot. Representative H2 frames show the state before and after the orientation repair. Every rollout writes into one identity-addressed trace, so all three kinds of evidence share a single result representation. and physical execution was left outside the empirical scope. DeepInsight I keeps that substrate fixed and closes the empirical gap. Table 1 states the delta. Contributions. This paper contributes evaluation infrastructure and empirical evidence, not new navigation, manipulation, or control policies: native benchmark protocols remain intact behind adapters, while execution, resource management, trace identity, and cross-domain reduction are shared. The rest of the paper follows the three empirical extensions in Table 1. Section 4 tests native-protocol fidelity across six System 1 benchmarks with released checkpoints. Section 5 freezes a common System 0 contract, compares four released WBC models in simulation, and carries a qualified cohort into matched real-robot trials. Section 6 extends shared-trace localization into handoff-specific attribution, a measured repairability criterion separating interface-repairable boundary errors from action-space limitations, and physical episodes that test the same attribution under hardware-observable state. 2 Related Work Embodied benchmarks. The benchmarks this paper brings onto one runtime are drawn from three ecosystems, and we survey them as the workload to be carried rather than as alternatives to compete with. In navigation, the instruction-following line (R2R [7], extended to continuous environments by VLN-CE [8] and to path-fidelity metrics by RxR [9]) supplies the workloads and metric suites this paper carries within the Habitat simulator family. In manipulation, LIBERO’s clean suites [3] are extended by perturbation and generalization variants such as LIBERO- Plus [10] and LIBERO-PRO [11]. Other benchmarks cover real-to-sim transfer, dual-arm control, mobile household tasks, long-horizon evaluation, and scene-rich environments [12–17]. At whole-body control, HumanoidBench [18] and the Isaac Lab ecosystem [4] anchor GPU-parallel locomotion evaluation. The motion-tracking controllers this paper actually evaluates descend from the motion-imitation line [19,20]: each released tracker—SONIC [21], HoloMotion [22]—ships its own evaluation protocol, with success predicates, tracking-error conventions, and reference-motion sets that do not interchange across releases; on the training side, Athena-WBC [23] argues that long-tail tracking failures reflect a mismatch between motion demands and learned capability rather than raw exposure. MotionBench (Section 5) freezes one such contract across trackers, with capability-grouped reporting 2 Table 1. Scope delta from DeepInsight v1 [6] to DeepInsight I. The core substrate is inherited unchanged; DeepInsight I is an empirical and execution-domain extension of v1, not a replacement of its architecture. DimensionDeepInsight v1 establishedDeepInsight I adds Core substrateTask, resource, and result: onereset/stepdriver, oneacquire/releaseprotocol, one identity-addressed trace. No new abstraction; embodied benchmarks and a physi- cal robot bind to the same substrate. System 2Quantitative fidelity, throughput, ablations, and multi- node scaling. Inherited; not re-benchmarked here. System 1Simulation-only coverage case studies. Native-protocol fidelity: released checkpoints on two navigation and four manipulation benchmarks. System 0Trace-backed screening of internal policies, in simulation only. MotionBench: four released WBC models under one frozen contract. Simulation to hardwareOutside the empirical scope.Matched sim and real-robot trials under one parent trace identity, with domain-specific records. Full-system diagnosisTask-, subgoal-, and subsystem-level failure localization, in simulation. Five handoff labels, each mapped to a repair; measured repairability; tested on physical episodes. aimed at the same axis. DeepInsight preserves each benchmark’s protocol—suites, splits, embodiment, success criteria—and unifies the runtime, resource layer, and trace beneath them; the fragmentation documented above is, in effect, this paper’s workload specification. Running embodied benchmarks. Most of these benchmarks are run through the native harness each one ships. The closest infrastructure effort to ours at System 1 is the VLA evaluation harness of Allen Institute for AI[5], which decouples models from benchmarks by running each benchmark in its own container against a GPU-resident model server over a WebSocket message protocol. Batched inference, task sharding, and result aggregation ride on that separation, dissolving the dependency conflicts that make embodied benchmarks notoriously non-co-installable. We adopt the same model-server view of the policy under test in our resource binding (Section 3); beyond such a federation, the shared runtime adds one episode driver, one result identity across benchmarks, and a real robot attached behind the same resource protocol—the distinction between coordination and a shared runtime remains, as in v1 [6], structural rather than nominal. Isaac Lab [4] supplies parallel-physics infrastructure that our System 0 workloads build on, and RoboArena [24] makes distributed real-world evaluation a first-class venue; neither, by design, joins matched simulated and physical rollouts under one trace identity, which is the property our sim-to-real study depends on. Measuring the sim-to-real relation. A recent line establishes the predictive validity of simulation for real-robot evaluation—pairing simulated and physical rollouts, comparing model orderings and matched errors, and arguing that simulation-based estimators suffice for checkpoint selection [12,25–28]. A complementary construction line narrows the gap rather than measuring it, rebuilding simulator scenes from sparse real captures around a robot-executable proxy [29]. We inherit from the measurement line rather than compete with it: the correspondence analysis of Section 5 descends from its metric families, and its qualification logic shapes our hardware admission gate. What changes is the domain and the substrate: the line has centered on manipulation policies and locomotion checkpoints, whereas Section 5 asks its question for whole-body motion-tracking controllers—and matched simulation and hardware branches share one trace identity and evaluation contract, so the correspondence evidence is produced by the runtime rather than reconstructed in a bespoke pipeline beside it. 3 Same Abstractions, New Backends Bringing embodied evaluation onto the DeepInsight substrate—up to and including a physical robot—is a change of backends, not of architecture. The three v1 invariants enter unchanged: every episode, whatever its shape, is driven through onereset/stepinterface with all transient state on a per-episode handle; every expensive backend is reached only throughacquire/release, its deployment, scaling, and failure handling staying inside a control plane behind the handle; and every event any subsystem produces is written into one append-only trace, addressed by a hierarchical identity that carries causal lineage [6]. Nothing in these invariants says what sits behind a handle. Three new backend classes exploit that property; Figure 2 sketches how each attaches. Simulators enter as sandboxed backends. The simulators behind the embodied benchmarks—Habitat, Mu- JoCo/robosuite, SAPIEN/ManiSkill—are third-party stacks with mutually conflicting dependencies. Each ships as a container image behind the sandbox handle, adopting the containerized view of Allen Institute for AI[5], so benchmarks stay dependency-isolated while the plane’s lease lifecycle, parallelism, and accounting apply to them unchanged. 3 Embodied evaluation is expressed through the same three abstractions as v1 — task, resource, and result. A simulator, a policy, and a real robot all attach as resources behind one handle, and a single rollout leaves one identity-addressed trace. The two rollouts differ only in the embodiment field, so sim-to-real comparison is a join over traces rather than a reconciliation across toolchains. 1 task ABSTRACTION reset(handle) · step(handle, action) → obs, event, done 2 resource ABSTRACTION acquire() → handle · release(handle) Simulator parallel pool · retryable failures Policy enters like an LLM · one config field Real robot SAME HANDLE serial · human-gated reset · non-retryable 3 result ABSTRACTION one identity-addressed trace per rollout — every event under a single identity Simulated rollout embodiment = sim Physical rollout embodiment = real execution_domain = simulation execution_domain = real_robot Figure 2. The embodied evaluation substrate. Task contracts define the episode and benchmark semantics; simulators, policies under test, and the real robot attach through resource bindings; and every rollout writes into an identity-addressed result trace. Matched simulated and physical episodes share a parent trace identity while retaining execution-domain-specific records. The policy under test enters as an inference backend. A model server exposes an observation-in, action-out interface and registers into the same service catalog as a language model, so substituting the policy under test is a configuration change, not an integration. Each benchmark binding declares the observation and action interface it expects from the policy server; the declaration is validated when benchmark and policy are bound, so an interface mismatch fails at configuration time rather than mid-rollout. The real robot enters as one more backend—asymmetric, but a backend nonetheless. The invariants extend unchanged: the robot is leased per episode, the lease is bound to the episode identity, and every event it produces writes into the shared trace. The physical backend’s asymmetries—serial capacity where a simulator pool is parallel, resets that may be human-gated, and interventions and emergency stops that are physical outcomes rather than retryable infrastructure errors—are absorbed as backend machinery behind the handle. Concretely: a safety interlock sits in front of the command stream; human-gated resets are logged as trace events; interventions and emergency stops are recorded as first-class outcomes rather than out-of-band incidents; and binding each episode to a specific robot unit keeps hardware variance decomposable. Above the handle, a real-robot episode is acquired, driven, and released like any other. One trace, two execution domains. On the trace, a matched pair of rollouts is organized under a single parent trace identity: the simulated and physical episodes hang as sibling branches beneath it, each retaining its own execution- domain-specific record and discriminated by anexecution_domainfield (simulationorreal_robot). What makes the pair comparable is recorded rather than assumed: the pairing key refers to a measured initial-state calibration—the robot’s starting configuration is measured, the simulator is initialized from those measurements, and any residual misalignment is retained as a covariate of the pair. The matched rollouts share the same task and metric contract while retaining execution-domain-specific episode records, so cross-domain reduction is a join on the parent identity rather than a reconciliation across result formats. The result is that “the same evaluation, in simulation and on hardware” is a well-defined phrase on this substrate: one driver, one evaluation contract, and one hierarchical trace. 4 System 1: Benchmark Fidelity and Common-Protocol Evaluation This section evaluates released System 1 checkpoints through the shared DeepInsight runtime. Benchmark-fidelity experiments retain the source task split, reset distribution, success predicate, observation/action interface, and aggregation rule. Where a controlled cross-model comparison requires an aligned setting, the shared configuration 4 and its differences from the source protocols are stated explicitly. A published value is used as a reference (Ref.) only when it can be tied to an identified checkpoint and protocol revision; DI denotes our execution. When a source publishes only aggregate values, the comparison is aggregate rather than episode-paired. Models without a verifiable reference are reported as DI-only. Table 2 maps the coverage: two navigation benchmarks (Section 4.1) and four manipulation benchmarks (Sec- tion 4.2), each entering through the shared driver and resource bindings of Section 3, with released checkpoints as the systems under test. The remainder of the section reports the resulting reference comparisons. 4.1 Navigation Unified evaluation configuration. We evaluate R2R-CE and the English RxR-CE val-unseen split in Habitat-Sim using 1,839 and 3,669 episodes, respectively. Every DI run uses the same episode list, an egocentric monocular camera as the policy input, and a maximum horizon of 500 navigation steps. Privileged simulator state is used only by the environment and metric implementation and is never exposed to the policy. The models share the same DeepInsight episode driver, resource binding, and trace schema; only checkpoint-specific preprocessing and action decoding remain inside the model adapter. R2R-CE emphasizes English instruction following and stopping in continuous scenes, whereas RxR-CE contains longer routes and denser instruction–trajectory grounding [8,9,30]. Figure 3 illustrates one R2R-CE episode under this configuration. The evaluated cohort consists of the released StreamVLN, RynnBrain-Nav, and JanusVLN checkpoints [31–33]. The Ref. columns provide the values reported by the source publications, while all DI columns use the same aligned 500-step configuration and form the controlled cross-model comparison. Figure 3. Example R2R-CE episode from the unified navigation evaluation. Six semantic keyframes show the egocentric monocular observations used for policy inference alongside the trace-derived trajectory visualization, instruction, and terminal metrics. The top-down map is used only for visualization and is not exposed to the policy. Metrics. Success rate (SR) measures correct task completion, while success weighted by path length (SPL) discounts successful but inefficient trajectories [34]. Navigation error (NE) is the final geodesic distance to the goal. R2R-CE additionally reports oracle success rate (OSR), which credits an episode if any visited state enters the success region. RxR-CE instead reports normalized Dynamic Time Warping (nDTW), which measures alignment between the executed and reference trajectories [35]. Bounded metrics are reported on a0–100scale; NE is in meters. Reproduction fidelity and residual differences. Overall, the DI results remain close to the published references and preserve their metric scale and qualitative behavior. On R2R-CE, StreamVLN differs by at most 0.55 percentage points across the bounded metrics, RynnBrain-Nav by 2.05 points, and JanusVLN by 3.81 points; their NE differences are 0.18, 0.10, and 0.14 meters, respectively. The RxR-CE reproduction is similarly close: StreamVLN matches the reported SR exactly, while the bounded-metric differences for all three checkpoints are at most 1.27 5 Table 2. System 1 coverage: six benchmarks, two domains, one substrate. Models are released checkpoints; reported metrics combine benchmark scores and trace-derived execution diagnostics and are reported in Table 3–Table 7. DomainBenchmarkSimulatorModels (released checkpoints)Reported metrics Navigation R2R-CEHabitat-SimStreamVLN, RynnBrain-Nav, JanusVLNSR, SPL, OSR, NE RxR-CEHabitat-SimStreamVLN, RynnBrain-Nav, JanusVLNNE, SR, SPL, nDTW Manipulation LIBEROMuJoCo/robosuiteπ 0 ,π 0.5 ,π 0 -FAST, OpenVLA, OpenVLA- OFT per-suite SR LIBERO-PlusMuJoCo/robosuitesame checkpoints as LIBEROper-perturbation SR SimplerEnv (WidowX/Bridge) SAPIEN/ManiSkillStarVLA (3 heads)SR RoboCasa365MuJoCo/robosuiteDiffusion Policy,π 0 ,π 0.5 , GR00T N1.5split-level SR Table 3. Navigation results on val-unseen splits. All runs use egocentric monocular input, and all DI values use a maximum of 500 steps. For each metric, Ref. (gray) is the value reported by the corresponding source publication and DI is our unified execution. Bounded metrics are percentages; NE is in meters. (a) R2R-CE, 1,839 episodes. Model SR↑SPL↑OSR↑NE↓ Ref.DIRef.DIRef.DIRef.DI StreamVLN [31]56.4055.8550.2049.9763.6063.134.905.08 RynnBrain-Nav [32]58.6057.3149.6048.0171.6069.554.925.02 JanusVLN [33]60.5057.2656.8052.9965.2062.914.784.92 (b) RxR-CE English, 3,669 episodes. Model SR↑SPL↑nDTW↑NE↓ Ref.DIRef.DIRef.DIRef.DI StreamVLN [31]54.4054.4045.4045.8163.7064.765.655.60 RynnBrain-Nav [32]56.1056.8542.7043.9759.6059.206.205.57 JanusVLN [33]56.2055.3647.5046.5662.1061.626.065.69 points and the NE differences are at most 0.63 meters. These results indicate that the shared runtime largely preserves the readings of the original evaluation stacks. The remaining gaps are consistent with several protocol- and implementation-level effects. First, the source publications provide aggregate results rather than episode-level outputs with matched random seeds, so Ref.–DI comparisons cannot control simulator stochasticity and episode ordering exactly. Second, residual differences in software revisions and at the adapter boundary—including image preprocessing, action decoding, collision handling, and stopping implementation—can change an early motion and then compound through closed-loop execution. This accumulation is especially visible on the longer RxR-CE trajectories, where modest per-step deviations can alter both endpoint metrics and nDTW without indicating a systematic failure of the reproduced policy. All DI results nevertheless use the same episode lists, monocular observations, and 500-step limit, so these residual gaps are not caused by unequal evaluation budgets. 4.2 Manipulation Benchmark coverage. We ask whether one evaluation substrate can retain native benchmark behavior across changes in simulator, embodiment, and task structure. The selected benchmarks provide complementary tests: LIBERO supplies a clean-suite baseline for single-arm manipulation; LIBERO-Plus adds seven controlled pertur- bation families; SimplerEnv moves the evaluation to SAPIEN/ManiSkill and the WidowX/Bridge embodiment; and RoboCasa365 introduces mobile manipulation and compositional household tasks [3,10,12,14]. Together, they separate clean-suite execution, perturbation sensitivity, cross-backend and cross-embodiment portability, and compositional task evaluation. Evaluation setup. We evaluate identifiable, author-released or benchmark-distributed checkpoints from theπ, OpenVLA, StarVLA, Diffusion Policy, and GR00T families [36–43]. Each DI run retains the benchmark’s task split, reset distribution, observation and action interfaces, success criterion, and aggregation rule. LIBERO references come from the corresponding model releases, with OpenVLA-OFT using the official 4-in-1 checkpoint [44]. LIBERO-Plus reuses these checkpoints without post-training; because it provides noπ 0.5 reference, we use an independent study of the official checkpoint [45]. SimplerEnv references are the StarVLA model-zoo aggregates. 6 StarVLA-OFT is excluded because the released Bridge checkpoint was confirmed to be incorrect and no corrected artifact was available [46]. RoboCasa365 uses official leaderboard references; all DI runs use v1.0.1, and an effective-step audit verifies that the official1.5×horizon is applied once and aligned with the reference evaluations [47, 48]. Evaluation scale. LIBERO runs4suites× 10tasks× 50episodes, totaling 2,000 episodes per model; LIBERO- Plus evaluates 10,030 test instances once each. SimplerEnv runs four sweeps over the 24 official object-pose configurations for each of four WidowX tasks, totaling 384 episodes per model [12,41]; its reported average is the unweighted mean of the four task rates. RoboCasa365 runs 50 episodes on each of 50 tasks, totaling 2,500 episodes per model, split into 900 atomic-seen, 800 composite-seen, and 800 composite-unseen episodes. Table 4. LIBERO success rates across the four standard suites. For each suite, Ref. (gray) is the published value and DI is our execution. Model Spatial↑Object↑Goal↑LIBERO-10↑Avg.↑ Ref.DIRef.DIRef.DIRef.DIRef.DI π 0 96.80 97.80 98.80 97.40 95.80 92.80 85.20 81.40 94.15 92.35 π 0.5 98.80 98.80 98.20 98.40 98.00 97.20 92.40 91.80 96.85 96.55 π 0 -FAST96.40 97.00 96.80 97.20 88.60 87.20 60.20 62.20 85.50 85.90 OpenVLA84.70 86.20 88.40 86.80 79.20 77.80 53.70 54.00 76.50 76.20 OpenVLA-OFT (comb.) 96.80 98.00 97.00 97.80 95.40 96.40 98.00 95.40 96.80 96.90 Table 5. LIBERO-Plus success rates by perturbation family. Total is episode-weighted; Ref. (gray) is the published value and DI is our execution.†: the publishedπ 0 Sensor-Noise reference is an unresolved benchmark-reference issue (see text). Model CameraRobot init.Lang.LightBackgr.NoiseLayoutTotal↑ Ref.DI Ref.DIRef. DIRef. DIRef.DI Ref.DI Ref.DI Ref.DI π 0 13.8 17.0 6.07.5 58.8 64.9 85.0 86.9 81.4 83.3 79.0 † 18.7 68.9 73.3 53.6 46.8 π 0.5 75.4 71.9 77.5 74.0 85.6 85.6 96.9 96.7 94.6 95.0 89.7 86.8 85.7 86.7 85.7 84.3 π 0 -FAST65.1 63.1 21.6 22.8 61.0 69.5 73.2 75.6 73.2 74.5 74.4 72.0 68.8 73.8 61.6 63.6 OpenVLA0.80.53.53.7 23.0 37.5 8.1 13.7 34.8 36.3 15.2 5.9 28.5 42.2 15.6 19.2 OpenVLA-OFT (comb.) 55.6 55.9 21.7 24.1 81.0 85.2 92.7 93.3 91.0 87.4 78.6 73.8 68.7 74.0 67.9 68.7 Table 6. SimplerEnv WidowX/Bridge Visual-Matching success rates. The model zoo publishes only the four-task average, so per-task values are DI-only; for the average, Ref. (gray) is the published value and DI is our execution. Model Per-task success (DI)Avg.↑ Stack cubeCarrot/PlateSpoon/TowelEggplant/BasketRef.DI StarVLA-FAST41.745.875.087.558.662.5 StarVLA-π34.469.882.376.062.565.6 StarVLA-GR00T40.655.281.377.163.663.5 Table 7. RoboCasa365 multi-task-learning success rates by task split. For each split, Ref. (gray) is the official leaderboard value and DI is our execution. Model Atomic seen↑Comp. seen↑Comp. unseen↑Overall↑ Ref.DIRef.DIRef.DIRef.DI Diffusion Policy15.7016.560.200.131.250.136.106.04 π 0 34.6034.676.105.881.101.6314.8014.88 π 0.5 39.6041.677.106.131.202.1316.9017.64 GR00T N1.550.7049.0014.8016.252.704.1323.9024.16 Qualitative rollout evidence. The aggregate scores above reduce each episode to a terminal success value, whereas Figure 4 exposes the trajectory-level interaction that produces that outcome. Across four representative 7 successful cases, the observations evolve as policy actions modify the scene over benchmark-specific horizons. This dependence distinguishes VLA evaluation from single-turn VLM scoring: later policy inputs are consequences of earlier actions within the same environment-coupled rollout. The displayed frames are sparse samples and do not imply a shared phase structure across tasks. SAMPLED ROLLOUT FRAMES (time →) (a) LIBERO π0.5 Pick up the black bowl between the plate and the ramekin and place it on the plate. t = 0t = 40t = 46t = 72 SUCCESS (b) LIBERO-Plus π0.5 · zero-shot Pick up the black bowl between the plate and the ramekin and place it on the plate. t = 0t = 42t = 48t = 153 SUCCESS (c) RoboCasa365 π0.5 Open the cabinet doors. t = 0t = 30t = 90t = 314 SUCCESS (d) SimplerEnv StarVLA-PI Put eggplant into yellow basket. t = 0t = 6t = 12t = 120 SUCCESS Figure 4. Representative successful closed-loop rollouts from LIBERO, LIBERO-Plus, RoboCasa365, and Sim- plerEnv (top to bottom). Each row identifies the evaluated checkpoint and task and shows the reset observation (t = 0), two intermediate observations, and the final sampled observation at the indicated saved-rollout frame indices. Intervening observation–action steps are omitted; all four episodes satisfy their benchmark’s native success predicate. Overall results. Across standard LIBERO, SimplerEnv, and RoboCasa365, the DI aggregates track their published references without a systematic upward or downward shift across model families. Most LIBERO-Plus model– perturbation pairs follow the same pattern. The material exception isπ 0 under Sensor Noise: DI obtains 18.7, compared with the published reference of 79.0. This category accounts for most of theπ 0 Total difference. Independent evaluations using the public OpenPI checkpoint report the same 16–20% range, while the maintainers are auditing the provenance of the original reference result [49]. We therefore treat this entry (marked†in Table 5) as an unresolved benchmark-reference issue and exclude it from behavioral and model-ranking claims. Taken together, the quantitative comparisons and trajectory examples support the intended benchmark-level fidelity claim: the shared runtime does not introduce a systematic shift across the evaluated simulators, embodiments, and checkpoints, while localized discrepancies remain attributable to specific benchmark–reference pairs. 8 5 System 0: Scalable WBC Evaluation from Simulation to Real-Robot Evidence V1 used trace-backed simulation to screen and diagnose internal System 0 policies [6, §5.2]. This section turns that case study into a standardized evaluation and extends it in two directions: MotionBench freezes a common workload and metric contract across released WBC models, and matched real-robot execution turns simulation-only evidence into measured simulation–real correspondence. The frozen contract matters because WBC evaluation varies along three coupled axes—the motion workload, the WBC interface, and the execution domain—so evaluating each model through its own harness confounds model behavior with task semantics, interface conversion, and reduction rules. Here, each released model instead attaches through a versioned binding, scaling the same evaluation across architectures, checkpoints, and runtime interfaces while the workload, trace schema, and metric reducers stay fixed. Section 5.1 fixes the workload and metrics; Section 5.2 compares four released WBC models and qualifies hardware cases; and Section 5.3 evaluates the two SONIC variants in matched real-robot trials. Broad in simulation, grounded on hardware — one evaluation contract, one parent trace identity 1 MotionBench motion-tracking workload of WBC Bench 2 Simulation heterogeneous WBC models, one contract 3 Real robot matched trials on one Unitree G1 Motion capabilities upper body · locomotion · in-place · dynamic · low posture Curation retarget quality annotate screen frozen snapshot: Core1000, 1,000+ cases Evaluation contract taskWBC binding trace metric four versioned components, frozen up front; core outputs SR · MPJPE-l · progress-soft WBC models heterogeneous architectures and native interfaces one declared WBC interface single-policy or encoder / decoder bindings parallel envs × N models every model runs the whole workload; environments execute concurrently Hardware admission 1,000+ cases 25 matched clips admitted on 100% simulation SR plus trace-derived checks Gap = Real − Sim MPJPE-lj. posj. velacceljerk simreal same motion, reference, checkpoint and body set; three trials averaged per pair one evaluation contract one parent trace identity shared metric definitions separate episode records Figure 5. System 0 evaluation overview. MotionBench fixes the versioned workload and measurement semantics; heterogeneous WBC models are compared in a common simulation space; and trace-qualified WBC–motion pairs enter matched real-robot trials for simulation–reality analysis. SR is used only in simulation; hardware correspondence uses matched root-relative and joint-space measurements. 5.1 Standardized WBC Evaluation MotionBench is the motion-tracking workload of our internally developed WBC Bench. Here it provides a curated, versioned System 0 workload for evaluating released WBC models. Candidate references are retargeted to the evaluation robot profile, quality-assessed, manually annotated, and screened by difficulty and motion type. Each frozen snapshot assigns a stable identity and metadata to every case. For report-level aggregation, cases are grouped into five capabilities—upper body, locomotion, in-place, dynamic, and low posture—and can also be decomposed by duration and motion. The common evaluation contract is instantiated through four versioned components: the task component fixes the case and reference; the WBC binding component declares the model interface; the trace component specifies retained evidence; and the metric component fixes its interpretation. Replacing a WBC model changes only its registry record and binding; the workload, failure semantics, trace schema, and reduction remain fixed. The headline simulation metrics separate executability from fidelity. Throughout this section, SR is the fraction of motions that avoid a hard failure: root-height or end-effector-height error above 0.25 m, or root-orientation error above 1.0 rad [21]. MPJPE-l is root-translation-subtracted mean per-body position error without rotational alignment; it is stored in millimeters and reported in centimeters. Progress-soft is the macro-average of each motion’s completion ratio before a hard failure or global position/heading drift. Together with the workload and versioned contracts above, these metrics define the System 0 evaluation specification summarized in Table 8. 9 Table 8. MotionBench specification. The benchmark fixes the workload and evaluation semantics before model- specific execution. ComponentSpecificationEvaluation role SnapshotFrozen MotionBench snapshot (Core1000; 1,000+ cases)Stable case-level comparison CurationRobot-profile retargeting, quality assessment, manual annotation, and difficulty/type screening Controlled motion workload CoverageUpper-body, locomotion, in-place, dynamic, and low-posture mo- tion across multiple durations Capability and duration decomposition WBC bindingVersioned checkpoint and declared WBC interface under the com- mon evaluation contract Comparable execution across WBC models Core outputsSR, MPJPE-l, and progress-softExecutability, local fidelity, and usable com- pletion 5.2 Unified Simulation Evaluation of Heterogeneous WBC Models We evaluate four released WBC models with heterogeneous architectures and runtime interfaces on the same MotionBench snapshot. HoloMotion uses a causal decoder-only Transformer with sparsely activated MoE layers, whereas SONIC uses an MLP encoder–decoder architecture with an FSQ token representation [21,22]. In our evaluation, HoloMotion enters through a single-policy binding, whereas SONIC uses separate encoder and decoder bindings. All models share the workload, failure semantics, trace schema, and reducers. Figure 6 presents SR and MPJPE-l as complementary measures of executability and local fidelity, with exact aggregates reported in Table 9. HoloMotion 1.3.2 leads the full benchmark on both measures, with 96.7% SR and 4.53 cm MPJPE-l; it also achieves the highest progress-soft value (93.1%). 88909294 96 98 4.5 5 5.5 6 HoloMotion 1.3.2 HoloMotion 1.4.0 SONIC release SONIC low-latency SR (%) MPJPE-l (cm) Compared WBC model Simulation leader Figure 6. Simulation SR–MPJPE-l plane on MotionBench. Each point is one released WBC model evaluated under the same task and metric contract. Table 9. MotionBench simulation results. SR and progress-soft are percentages; MPJPE-l is the macro average over motions. WBCArchitectureSR↑Progress-soft↑MPJPE-l (cm)↓ HoloMotion 1.3.2Sparse-MoE causal Transformer96.793.14.53 HoloMotion 1.4.0Sparse-MoE causal Transformer91.977.95.15 SONIC releaseMLP encoder–decoder + FSQ90.872.75.65 SONIC low-latencyMLP encoder–decoder + FSQ89.267.45.73 The capability decomposition in Table 10 localizes the aggregate differences. Upper-body progress-soft is nearly saturated across models, whereas locomotion, dynamic motion, and low-posture motion produce larger separation. The SONIC variants achieve hard-failure SRs of 90.8% and 89.2%, while their progress-soft values are 72.7% and 67.4%, respectively, revealing a larger separation under a metric that accounts for partial completion and global drift. HoloMotion 1.4.0 shows its largest deficits on locomotion and dynamic motion. For the matched real-robot study, we predeclare SONIC release and SONIC low-latency as the hardware cohort. They share the same architecture and WBC interface, enabling a release-level comparison under the same protocol. 10 Table 10. MotionBench progress-soft (%) by capability. The decomposition localizes losses in usable tracking behind aggregate SR. WBCUpper bodyLocomotionIn-placeDynamicLow posture HoloMotion 1.3.297.795.191.490.482.5 HoloMotion 1.4.095.271.083.154.368.0 SONIC release93.863.474.755.755.6 SONIC low-latency93.852.570.548.450.8 5.3 Grounding Simulation Findings in Real-Robot Evidence Hardware qualification. Before hardware execution, we freeze a shared subset of 25 motion clips for the two SONIC variants. A WBC–motion pair is admitted after achieving 100% simulation SR and passing trace-derived checks of trajectory completion, local tracking, root and end-effector posture, and actuator/contact behavior. Qualified pairs are executed three times on one Unitree G1 from the same initialization protocol. Simulation SR serves as the qualification metric; hardware evaluation uses root-relative pose and proprioceptive joint-space measurements. Matched protocol. Each simulation rollout and its three hardware trials share a parent trace identity and bind the same motion, reference, checkpoint, adapter version, body set, and metric definitions, while retaining separate episode records. Hardware MPJPE-l is computed from synchronized robot-state estimates and forward kinematics using the same root-translation subtraction as in simulation. Joint-position and joint-velocity RMSE use the matched reference trajectory; acceleration and jerk statistics use the executed joint trajectories. The three hardware trials are averaged for each WBC–motion pair, followed by an average over matched motions. Simulation–real correspondence. Table 11 reports the matched kinematic and dynamic profile of both SONIC variants. MPJPE-l measures root-relative pose fidelity; joint-position and joint-velocity RMSE measure reference tracking; and acceleration and jerk RMS/P95 summarize sustained and upper-tail joint dynamics. The simulation entries are recomputed on the same 25-clip subset and therefore differ from the full-MotionBench aggregates in Table 9. Table 11. Matched simulation–real kinematic and dynamic profile for the qualified SONIC cohort over 25 matched motion clips. For each variant, Sim. (gray) and Real give the paired aggregates and Gap is(Real− Sim)/Sim× 100%. Hardware values average three trials per WBC–motion pair before aggregation across motions. Metric SONIC releaseSONIC low-latency Diagnostic role Sim.RealGap (%)Sim.RealGap (%) MPJPE-l (cm)3.083.41+10.833.283.89+18.65Root-local whole-body tracking Joint-position RMSE (rad)0.130.16+22.150.150.18+22.53Joint pose tracking and steady bias Joint-velocity RMSE (rad/s)0.740.75+1.580.800.88+10.46Dynamic tracking, latency, and damping Joint-acceleration RMS (rad/s 2 )18.4014.16 −23.0522.3220.39 −8.66Sustained high-frequency activity Joint-jerk RMS (rad/s 3 )1181.51941.36 −20.331442.24 1375.07 −4.66Sustained command discontinuity Joint-acceleration P95 (rad/s 2 )27.0127.69+2.5232.9539.07+18.58Transient acceleration peaks Joint-jerk P95 (rad/s 3 )1518.76 1770.80+16.601854.33 2517.01+35.74Sharp spikes and actuator stress SONIC release remains lower than SONIC low-latency across all seven readings in both domains. Their joint- position gaps are similar, whereas low-latency shows larger MPJPE-l (+18.65% versus +10.83%), joint-velocity (+10.46% versus +1.58%), acceleration-P95 (+18.58% versus +2.52%), and jerk-P95 (+35.74% versus +16.60%) gaps. For both variants, hardware increases tracking errors and tail peaks while reducing acceleration/jerk RMS. Acceleration and jerk gaps characterize motion dynamics rather than independent success criteria. 6 Full-System: Handoff Attribution and Repairability in Physical Execution V1 localized composed System 2–1–0 failures to the task, subgoal, and subsystem levels over the shared trace [6]. DeepInsight I adds a second automatic stage: when the localized failure falls on a system handoff, attribution identifies the specific cross-module assumption, interface convention, or coordination design that broke, and maps it to a repair action. Every decision must be supported by a concrete trace interval. We evaluate the mechanism in simulation and on a real robot. Attribution sorts handoff failures into two causes whose repairs cannot substitute for each other: 11 •Boundary-definition problem. The upstream termination criterion and the downstream precondition are not defined over the same state space, the two criteria evaluate different quantities, or a necessary condition was never registered in the interface. Adding the missing interface convention repairs it. • Executability problem. The boundary is correct, but the downstream feasible entry region is comparable in width to the minimum executable step, so corrections cannot land inside it reliably—and cannot at all once the region is narrower than one step. No interface convention repairs this; the action space must change. Separating the two converts handoff failure from an undifferentiated label into an actionable repair direction. 6.1 Automatic Attribution of Handoff Failures We reuse the object-introduction task of the first report—find and greet the user, resolve a referential introduction request, take up a pose from which the object (here a vehicle) can be presented, present it, and terminate— whose chain contains six handoffs, H1–H6 (Table 14). Each skill is first run from controlled initial states to register its contract—termination signal, start condition, observable postcondition—making implicit handoff assumptions testable. Analysis concentrates on the alignment handoffs H2 (navigation→greeting) and H5 (navigation→ presentation), whose geometric preconditions admit a quantitative repairability judgment. Attribution checks each state declared at the localized boundary against an independently reconstructed one—rules evaluate geometric and controller predicates, a multimodal judge the semantic and visual ones—and yields five labels, each mapped to one repair action (Table 12); when evidence is insufficient, attribution abstains rather than forcing a label. The declared–reconstructed split is not redundancy: embodied agents’ terminal self-reports are known to diverge from their verified world state [50], and a termination error is that divergence surfacing at a handoff boundary. Simulation additionally records privileged pose, collision state, and multi-view video around each handoff, used only to supervise and audit the diagnosis. Table 12. The five attribution labels: each verified evidence pattern maps to one repair action. The first three apply at the handoff boundary, the last two inside the downstream skill’s execution; abbreviations are used in Table 14. Evidence patternLabelRepair Source reports success; its verified postcondition failsTermination error (TE)change the termination criterion Source postcondition holds; a verified target precondition fails (readiness error if the target also signaled ready) Handoff-condition violation (HCV) add the missing precondition or alignment criterion Both modules report success and all registered conditions pass; the composed step still fails Contract gap (CG)register the omitted condition, unless the geometry check shows the entry region is execution-limited Entry handoff verified, preconditions hold throughout; failure inside the skill’s execution interval Skill-internal execution (SIE) improve the invoked skill; interface untouched A condition that held at entry lapses mid-execution (user walks away, occlusion appears) Execution-time invariant lapse (EIL) promote the condition to continuous monitoring Two causes and a repairability criterion. Regrouped by repair action, the labels give the main result of this section. H2 is the representative boundary-definition problem:navigate_to_targetterminates on position alone while the greeting requires facing the user, so the handoff breaks by construction—8 of 10 isolated attempts failed before the orientation criterion was added 1 —and the repair is to add the missing interface convention. H5 is the representative executability problem: every condition is registered and passes, yet entry keeps failing, because geometry binds against actuation. The criterion measures the feasible entry region in units of the minimum executable step. If the region is narrower than one step, no correction can stop inside it and the handoff is structurally unrepairable in the current action space; if wider, correction is reliable only when the leftover slack clearly exceeds execution error. Both quantities are measured—the step from the actuator, the region from the downstream skill’s rollout success surface—so the criterion is decidable before execution. It also disambiguates a contract gap, whose evidence pattern admits both causes: ample slack means a condition was omitted, slack on the order of the execution error means the entry region is execution-limited. Both handoffs share the same0.20m step: H2’s0.40m region leaves0.20m of slack and the orientation criterion repairs it, whereas field of view and occlusion squeeze H5’s to0.30m and the remaining 0.10m is the same order as the execution error. Table 13 confirms this with a local adjustment budget (corrective steps allowed after upstream termination to reach the downstream start pose): at H2 a single step nearly saturates readiness, whereas H5 starts far lower and needs the full five-step budget to catch up, each extra step re-rolling the same error against the same slack. Figure 7 draws the two states at the end of navigation. 1 Counts from an isolated H2 test, not from the composed episodes of Table 14. 12 H2 navigation→ greeting end of navigation navigation: 1 m, any heading user step 0.20 m 45° greeting: 0.40 m band both modules: success standoff floats±0.2 m and still qualifies; the 45° is never registered, so nothing asks for it HCV → register an orientation condition H5 navigation→ presentation end of navigation navigation: 1 m, any heading vehicle step 0.20 m user 150° 0.3 m outside presentation: 0.30 m band both modules: success the adjustment is declared — but the required start pose is tight, and one step spans the whole band CG → no condition to add; change the action space Figure 7. The two causes at the end of navigation. At H2 the heading error is never registered; at H5 the required start pose is declared but sits150 ◦ away, in a band one step wide. Angles and offsets are representative; band widths and the step are measured (Table 13). Table 13. Repairability criterion and readiness under a local adjustment budget. The entry region (rollout success surface) and the minimum step (actuator) are measured, and hence so is their ratio; Ready counts episodes whose handoff state lands inside the calibrated entry region, not those whose registered preconditions merely pass—at a contract gap the two differ. HandoffAxisRegion (m)Step (m)Steps wide Ready at budget 015 H2 navigation→ greetingapproach0.400.202.012/2016/2017/20 H5 navigation→ presentationapproach0.300.201.56/1511/1513/15 Simulation study. We execute 20 composed episodes under varied start poses, approach directions, referent ambiguity, occlusion, and user-confirmation behavior, at an adjustment budget of one step (the budget-1 column of Table 13), and compare automated attribution against human reference labels informed by privileged simulator state. Table 14 resolves the chain handoff by handoff: the failure mass and the hidden subset—failures in which both adjacent modules self-report success—concentrate on the two alignment handoffs. Table 14. Per-handoff breakdown of the composed episodes: episodes arriving at each handoff (Reached), breaking there (Failed), and failures with both adjacent modules self-reporting success (Both OK). Label abbreviations follow Table 12; Abst is abstention; dashes denote zero. HandoffReachedFailedBoth OK Attributed label TEHCVCGSIEEILAbst H1 retrieval→ nav. (person)20— H2 nav.→ greeting204213— H3 greeting→ request1610—1— H4 resolution→ nav. (viewpoint)15— H5 nav.→ presentation1543—31— H6 presentation→ termination1111—1 Total (20 episodes, 10 completed)—106133111 A negative control confirms discrimination: one H5 failure passes the entry check like the contract gaps around it, but the trace shows the break occurring mid-execution on the safety axis rather than pose, and attribution assigns it to skill-internal execution, not a boundary. 6.2 Real-Robot Evidence We run the same task on the physical robot, with the diagnostic restricted to hardware-observable inputs—module and tool status, estimated robot state, controller events, onboard video, and odometry pose—and human reference 13 labels from the aligned trace, onboard video, and on-site observation. Across 8 episodes, automated labels agree with the reference on task outcome in every episode and on handoff attribution in 2 of the 3 failed-or-partial ones, and every failure cause maps onto a class already identified in simulation. Separately, in one episode the user interrupted mid-skill without causing failure: control transfers outside all declared handoffs, for which the interface defines no preemption semantics, and responding would require duplex interaction—left to future work. One case shows the diagnostic resolving what module status alone cannot: episode 6 fails at H2 with both modules self-reporting success, and attribution returns a handoff-condition violation, citing [t=39.3s,t=49.1s]. Odometry drift, which hardware permits us to bound only to0.10–0.20m, enlarges the correction the robot must actually apply, leaving at most0.10m of the0.20m slack that the one-step adjustment relies on in simulation—no more than the margin at which H5 already fails. Figure 8 shows the state this resolves: the declared evidence is the same whether the handoff breaks or completes. H2 nav.→ greetingepisode 6 — failed HCV → register an orientation condition drift enlarges the correction one step H2 nav.→ greetingcompleted none → the omission is not exercised the missing condition is unchanged both executions declare the same: modules report success, heading never registered — the declaration cannot tell them apart Figure 8. H2 on hardware at the end of navigation. The dashed line marks the heading the greeting requires. Frames are representative; the entry region and step of Table 13 are calibrated in simulation, not re-measured on hardware. 7 Conclusion We extended DeepInsight from a simulation-only substrate that quantified only the foundation-model layer into one that quantifies the embodied layers—navigation and manipulation (System 1) and whole-body control (System 0)— and carries a real robot behind the same handle. The same three abstractions absorbed embodied heterogeneity without a fourth. The distinctive return is that matched simulated and physical rollouts share a parent trace identity while retaining execution-domain-specific evidence. In System 0, this carries a standardized WBC comparison from scalable simulation to matched physical evidence: the two SONIC variants preserve their numerical ordering across seven matched readings, while hardware increases tracking error and shifts joint dynamics from lower RMS activity to larger upper-tail peaks. At the full-system level, aligned trace evidence also turns implicit skill-boundary assumptions into inspectable contracts. In the simulated case study, five-way attribution maps failures to concrete repair actions, and the measured ratio between a feasible entry region and the minimum executable step separates interface-repairable boundary errors from action-space limitations; on the physical robot, the same diagnosis reproduces the failure classes identified in simulation and shows odometry drift consuming the slack that the criterion budgets. References [1] Jialian Li, Junhong Liu, Yuchen Cao, Weiran Guo, Jiaming Song, Xutao Wang, Yi Zhao, Jiangpin Liu, and Jie Chen. Athena-Brain technical report: An efficient robot brain for general intelligence and embodied interaction. arXiv preprint arXiv:2607.18985, 2026. [2] Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, et al. Habitat: A platform for embodied ai research. In ICCV, 2019. [3]Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. In NeurIPS Datasets and Benchmarks, 2023. [4] Mayank Mittal et al. Orbit / isaac lab: A unified simulation framework for interactive robot learning. In IEEE RA-L, 2023. [5]Allen Institute for AI. Vla evaluation harness: Unified evaluation for vision-language-action models.https: //github.com/allenai/vla-evaluation-harness, 2025. [6]Siyi Li, Chunyu Sun, Jiahao Zhang, Yuchen Kang, Wuliang Wang, Yu Qiu, Rui Jiang, Haitao Cui, and Jie Chen. Deepinsight: A unified evaluation infrastructure across the physical ai stack. arXiv preprint arXiv:2606.17574, 2026. [7]Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In CVPR, 2018. 14 [8]Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, and Stefan Lee. Beyond the nav-graph: Vision- and-language navigation in continuous environments. In ECCV, 2020. [9]Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. Room-across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding. In EMNLP, 2020. [10]Senyu Fei, Siyin Wang, Junhao Shi, Zihao Dai, Jikun Cai, Pengfang Qian, Li Ji, Xinzhe He, Shiduo Zhang, Zhaoye Fei, Jinlan Fu, Jingjing Gong, and Xipeng Qiu. LIBERO-Plus: A progressive robustness benchmark for vision-language-action models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2026. [11]Xueyang Zhou, Yangming Xu, Guiyao Tie, Yongchao Chen, Guowen Zhang, Duanfeng Chu, Pan Zhou, and Lichao Sun. Libero-pro: Towards robust and fair evaluation of vision-language-action models beyond memorization. arXiv preprint arXiv:2510.03827, 2025. [12] Xuanlin Li, Kyle Hsu, Jiayuan Gu, et al. Evaluating real-world robot manipulation policies in simulation. In CoRL, 2024. [13] Tianxing Chen et al. Robotwin 2.0: A scalable data generator and benchmark with strong domain randomiza- tion for robust bimanual robotic manipulation. arXiv preprint arXiv:2506.18088, 2025. [14]Soroush Nasiriany, Sepehr Nasiriany, Abhiram Maddukuri, and Yuke Zhu. RoboCasa365: A large-scale simulation framework for training and benchmarking generalist robots. In International Conference on Learning Representations, 2026. [15] Oier Mees, Lukas Hermann, Erick Rosete-Beas, and Wolfram Burgard. Calvin: A benchmark for language- conditioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters, 2022. [16]Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J. Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 2020. [17]Soroush Nasiriany, Abhiram Maddukuri, Lance Zhang, Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Mandlekar, and Yuke Zhu. Robocasa: Large-scale simulation of everyday tasks for generalist robots. In RSS, 2024. [18]Carmelo Sferrazza, Dun-Ming Huang, Xingyu Lin, Youngwoon Lee, and Pieter Abbeel. Humanoidbench: Simulated humanoid benchmark for whole-body locomotion and manipulation. In RSS, 2024. [19] Zhengyi Luo, Jinkun Cao, Alexander Winkler, Kris Kitani, and Weipeng Xu. Perpetual humanoid control for real-time simulated avatars. In ICCV, 2023. [20] Chen Tessler, Yunrong Guo, Ofir Nabati, Gal Chechik, and Xue Bin Peng. Maskedmimic: Unified physics- based character control through masked motion inpainting. ACM Transactions on Graphics (SIGGRAPH Asia), 2024. [21]Zhengyi Luo, Ye Yuan, Tingwu Wang, Chenran Li, Fernando Castañeda, Sirui Chen, Zi-Ang Cao, Jiefeng Li, et al. SONIC: Supersizing motion tracking for natural humanoid whole-body control. arXiv preprint arXiv:2511.07820, 2026. [22]Maiyue Chen, Kaihui Wang, Bo Zhang, Xihan Ma, Zhiyuan Yang, Yi Ren, Qijun Huang, Zihao Zhu, Yucheng Wang, and Zhizhong Su. HoloMotion-1 technical report. arXiv preprint arXiv:2605.15336, 2026. [23]Yuan Jiang, Ningyuan Zhang, Xicun Yang, Yuzhi Jiang, and Jie Chen. Athena-WBC: Capability-aligned policy experts for long-tail humanoid whole-body control. arXiv preprint arXiv:2607.04837, 2026. [24] Pranav Atreya, Karl Pertsch, Tony Lee, Moo Jin Kim, Arhan Jain, et al. Roboarena: Distributed real-world evaluation of generalist robot policies. arXiv preprint arXiv:2506.18123, 2025. [25]Guangqi Jiang, Haoran Chang, Ri-Zhao Qiu, Yutong Liang, Mazeyu Ji, Jiyue Zhu, Zhao Dong, Xueyan Zou, and Xiaolong Wang. Gsworld: Closed-loop photo-realistic simulation suite for robotic manipulation. arXiv preprint arXiv:2510.20813, 2025. [26] Dylan Khor and Bowen Weng. Post-convergence sim-to-real policy transfer: A principled alternative to cherry-picking. arXiv preprint arXiv:2504.15414, 2025. [27]Jad Abou-Chakra, Lingfeng Sun, Krishan Rana, Brandon May, Karl Schmeckpeper, Niko Suenderhauf, Maria Vittoria Minniti, and Laura Herlant. Real-is-sim: Bridging the sim-to-real gap with a dynamic digital twin. arXiv preprint arXiv:2504.03597, 2025. [28] Filip Bjelonic, Fabian Tischhauser, and Marco Hutter. Towards bridging the gap: Systematic sim-to-real transfer for diverse legged robots. arXiv preprint arXiv:2509.06342, 2025. [29]Shuai Fang, Xin Deng, Yuchen Kang, Zhenjiang Li, and Jie Chen. R2S-EGO: Dual-proxy refinement for sparse-capture real-to-sim. arXiv preprint arXiv:2608.06827, 2026. [30] Yicong Hong, Zun Wang, Qi Wu, and Stephen Gould. Bridging the gap between learning in discrete and continuous environments for vision-and-language navigation. In CVPR, pages 15439–15449, 2022. [31]Meng Wei, Chenyang Wan, Xiqian Yu, Tai Wang, Yuqiang Yang, Xiaohan Mao, Chenming Zhu, Wenzhe Cai, Hanqing Wang, Yilun Chen, Xihui Liu, and Jiangmiao Pang. Streamvln: Streaming vision-and-language navigation via slowfast context modeling. In ICRA, 2026. [32]Ronghao Dang, Jiayan Guo, Bohan Hou, Sicong Leng, Kehan Li, Xin Li, Jiangpin Liu, Yunxuan Mao, Zhikai Wang, Yuqian Yuan, Minghao Zhu, et al. Rynnbrain: Open embodied foundation models. arXiv preprint arXiv:2602.14979, 2026. 15 [33]Shuang Zeng, Dekang Qi, Xinyuan Chang, Feng Xiong, Shichao Xie, Xiaolong Wu, Shiyi Liang, Mu Xu, and Xing Wei. Janusvln: Decoupling semantics and spatiality with dual implicit memory for vision-language navigation. In International Conference on Learning Representations, 2026. [34]Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, and Amir R. Zamir. On evaluation of embodied navigation agents. arXiv preprint arXiv:1807.06757, 2018. [35]Gabriel Ilharco, Vihan Jain, Alexander Ku, Eugene Ie, and Jason Baldridge. General evaluation for instruction conditioned navigation using dynamic time warping. In NeurIPS Visually Grounded Interaction and Language Workshop, 2019. [36] Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, et al.π 0 : A vision- language-action flow model for general robot control. arXiv preprint arXiv:2410.24164, 2024. [37] Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, et al.π 0.5 : A vision-language-action model with open-world generalization. arXiv preprint arXiv:2504.16054, 2025. [38] Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. FAST: Efficient action tokenization for vision-language-action models. arXiv preprint arXiv:2501.09747, 2025. [39]Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, et al. OpenVLA: An open-source vision-language-action model. In Conference on Robot Learning, 2024. [40] Moo Jin Kim, Chelsea Finn, and Percy Liang. Fine-tuning vision-language-action models: Optimizing speed and success. arXiv preprint arXiv:2502.19645, 2025. [41] StarVLA Community. StarVLA: A lego-like codebase for vision-language-action model developing. arXiv preprint arXiv:2604.05014, 2026. [42]Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Robotics: Science and Systems, 2023. [43]NVIDIA. GR00T N1.5: An improved open foundation model for generalist humanoid robots. Official model release, 2025. URL https://huggingface.co/nvidia/GR00T-N1.5-3B. [44] Moo Jin Kim. OpenVLA-OFT: Combined libero checkpoint. Official OpenVLA-OFT repository release, 2025. URL https://github.com/moojink/openvla-oft/issues/21. [45]Zhanguang Zhang, Zhiyuan Li, Behnam Rahmati, Rui Heng Yang, Yintao Ma, Amir Rasouli, Sajjad Pak- damansavoji, Yangzheng Wu, Lingfeng Zhang, Tongtong Cao, Feng Wen, Xingyue Quan, and Yingxue Zhang. Do world action models generalize better than VLAs? a robustness study. arXiv preprint arXiv:2603.22078, 2026. [46]Xiaoxi Sun. Qwen3-VL-Bridge-RT-1 checkpoint issues. GitHub issue #324 in the official StarVLA repository, 2026. URL https://github.com/starVLA/starVLA/issues/324. [47]RoboCasa Team. RoboCasa365 codebase and release notes. Official repository, 2026. URLhttps:// github.com/robocasa/robocasa. [48]RoboCasa Team. RoboCasa365 leaderboard. Official benchmark leaderboard, 2026. URLhttps:// robocasa.ai/leaderboard.html. [49]Skywalker-yqz. Reproducibility of the official OpenPIπ 0 sensor noise result on LIBERO-Plus. GitHub issue #61 in the official LIBERO-Plus repository, 2026. URLhttps://github.com/sylvestf/LIBERO-plus/ issues/61. [50] Ying Chen, Rui Jiang, Lihuang Fang, Mingxu Wang, Zhifeng Gu, Lei Yi, and Jie Chen. Done, but not sure: Disentangling world completion from self-termination in embodied agents. arXiv preprint arXiv:2605.08747, 2026. 16

Instant research discovery