Paper deep dive
MIKASA-Robo-VLA: Benchmarking Memory in VLA Models for Long-Horizon Manipulation
Egor Cherepanov, Nikita Kachaev, Aleksandr I. Panov, Alexey K. Kovalev
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 10/2/2026, 9:20:11 AM
Summary
The paper introduces MIKASA-Robo-VLA, a benchmark designed to evaluate memory capabilities in Vision-Language-Action (VLA) models for long-horizon manipulation tasks. It comprises 90 language-conditioned tasks across 10 distinct memory types (e.g., Object, Spatial, Tracking, Checklist), with 80 tasks requiring the agent to recall information that is hidden or occluded during the action phase. The benchmark provides 22,500 oracle trajectories in RLDS and LeRobotDataset v3 formats. A reference baseline using the pi_0.5 model, fine-tuned on 14 tasks without explicit memory modules, achieved a mean success rate of 0.211, highlighting the difficulty of long-horizon memory retention in current VLA architectures.
Entities (12)
Relation Signals (9)
MIKASA-Robo-VLA → contains → 90 language-conditioned manipulation tasks
confidence 100% · We introduce MIKASA-Robo-VLA, a benchmark of 90 language-conditioned manipulation tasks.
$\\pi_{0.5}$ → evaluatedon → MIKASA-Robo-VLA
confidence 95% · A reference $π_{0.5}$ baseline... is fine-tuned on 14 tasks and achieves 0.211 ± 0.044 mean task success.
MIKASA-Robo-VLA → implementsin → ManiSkill 3
confidence 95% · MIKASA-Robo-VLA comprises 90 language-conditioned manipulation tasks implemented in ManiSkill 3
MIKASA-Robo-VLA → releasesdatain → RLDS
confidence 95% · We release 22,500 oracle trajectories across 10 memory types in RLDS and LeRobotDataset v3.
MIKASA-Robo-VLA → releasesdatain → LeRobotDataset v3
confidence 95% · We release 22,500 oracle trajectories across 10 memory types in RLDS and LeRobotDataset v3.
MIKASA-Robo-VLA → extends → MIKASA-Robo
confidence 90% · We introduce MIKASA-Robo-VLA, an extension of MIKASA-Robo... to language-conditioned VLA training and evaluation.
Object → istypeof → MIKASA-Robo-VLA
confidence 90% · The suite covers 10 memory types (Table 1)... Object: The cue reveals which object identity...
Tracking → istypeof → MIKASA-Robo-VLA
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Vision-language-action policies often see only one or a few recent frames, which makes it difficult to evaluate how they use information that disappears during a task. We introduce MIKASA-Robo-VLA, a benchmark of 90 language-conditioned manipulation tasks. All but 10 hide the cue an action depends on. Those 10 are reactive controls. MIKASA-Robo, the suite it rebuilds, has 32 tasks and uses language only in a representative VLA subset. Here every task provides an instruction, while memory-dependent tasks hide a task-relevant cue and reactive controls keep it available. For 70 tasks, environment phase timings specify an information gap, and for 28 of them the gap exceeds the 16-frame window of the widest fixed-context VLA we survey. The gap counts only the interval the cue is provably absent, not the full duration a policy must retain it, so every memory-dependent task still requires memory by construction, including the ones whose measured gap is short. We release 22,500 oracle trajectories across 10 memory types in RLDS and LeRobotDataset v3. A reference $\pi_{0.5}$ baseline with current images and proprioception, but no observation history or explicit memory module, is fine-tuned on 14 tasks and achieves 0.211 $\pm$ 0.044 mean task success. Its lower success on the evaluated Long-split tasks is confounded by open-loop chunking and the memory types represented in that subset. Project page: this https URL
Tags
Links
- Source: https://arxiv.org/abs/2610.00604v1
- Canonical: https://arxiv.org/abs/2610.00604v1
Trouble viewing inline? Open PDF directly →
Full Text
163,152 characters extracted from source content.
Expand or collapse full text
MIKASA-Robo-VLA: Benchmarking Memory in VLA Models for Long-Horizon Manipulation Egor Cherepanov 1,2 Nikita Kachaev 1,3 Aleksandr I. Panov 1,2 Alexey K. Kovalev 1,2 1 AXXX, Moscow, Russia 2 MIRIAI, Moscow, Russia 3 HSE University, Moscow, Russia cherepanov@axxx.tech mikasarobo.github.io Object 18Spatial 14Capacity 12Temporal 12Negative 9 MIKASA-Robo-VLA A memory-intensive benchmark and dataset for vision-language-action models 90 language-conditioned tasks across 10 memory types 22,500 oracle episodes 14,000 MP + 8,500 PPO 6.3 M recorded frames 175 h at 10 Hz 25–2,160 step horizons 3 length splits Dense per-step reward · one instruction per episode, no privileged state · released in RLDS and LeRobotDataset v3 Sequential 6Procedural 6Prospective 5Tracking 4Checklist 4 ONE RECORDED EPISODE “Observe which color is under each cup, track the cups as they shuffle, then touch the cup matching the lamp color.” colors in viewcups cover › cups swap › lamp cues › arm commits › Here the colors are out of view for 24–31 steps before the lamp names one. The environment sets a hiding time in 70 tasks: 55 hide the cue longer than 1 frame, 28 longer than the 16-frame window of the widest- context VLA we survey. Longest gap, 1,000 steps. Abstract Vision-language-action policies often see only one or a few recent frames, which makes it difficult to evaluate how they use information that disappears during a task. We introduce MIKASA-Robo-VLA, a benchmark of 90 language-conditioned manipulation tasks. All but 10 hide the cue an action depends on. Those 10 are reactive controls. MIKASA-Robo, the suite it rebuilds, has 32 tasks and uses language only in a representative VLA subset. Here every task provides an instruction, while memory-dependent tasks hide a task-relevant cue and reactive controls keep it available. For 70 tasks, environment phase timings specify an information gap, and for 28 of them the gap exceeds the 16-frame window of the widest fixed-context VLA we survey. The gap counts only the interval the cue is provably absent, not the full duration a policy must retain it, so every memory- dependent task still requires memory by construction, including the ones whose measured gap is short. We release 22,500 oracle trajectories across 10 memory types in RLDS and LeRobotDataset v3. A referenceπ 0.5 baseline with current images and proprioception, but no observation history or explicit memory module, is fine-tuned on 14 tasks and achieves0.211± 0.044mean task success. Its lower success on the evaluated Long-split tasks is confounded by open-loop chunking and the memory types represented in that subset. Project page:mikasarobo.github.io. 40th Conference on Neural Information Processing Systems (NeurIPS 2026). Workshop: RoboPAD. arXiv:2610.00604v1 [cs.LG] 30 Sep 2026 1 Introduction Vision-language-action (VLA) models increasingly use pretrained visual and linguistic representa- tions to control robots from demonstrations. Models such as OpenVLA andπ 0.5 execute diverse manipulation skills and generalize across tasks and environments [Kim et al., 2024, Physical In- telligence et al., 2025]. Sustained interaction makes memory a central requirement: a robot must act on information that may no longer be visible. It may need to recall an object’s identity after an occlusion, reproduce a previously observed sequence, or track which steps of a task have already been completed. The instruction specifies the goal, but the observation history determines how to achieve it. Explicit-memory methods address this challenge in different ways. MemER retrieves relevant visual experiences for hierarchical control [Sridhar et al., 2025], while MEM combines visual and textual memory at different temporal scales [Torne et al., 2026]. Their physical setups make systematic memory sweeps costly to reproduce. Simulation offers a more scalable alternative, yet widely used suites such as CALVIN and LIBERO emphasize skill composition and knowledge transfer [Mees et al., 2022, Liu et al., 2023] rather than explicit control of what must be remembered and for how long. In complex multi-stage tasks, aggregate success also mixes memory demands with perception, planning, and motor execution. Memory-focused suites, including MemoryBench, LIBERO-Mem, RoboMME, and RoboMemArena, make these demands more explicit, but cover a narrower range of tasks and memory types [Fang et al., 2025, Chung et al., 2026, Dai et al., 2026, Lei et al., 2026]. We introduce MIKASA-Robo-VLA, an extension of MIKASA-Robo [Cherepanov et al., 2026a], orig- inally developed to evaluate memory in reinforcement-learning (RL) agents, to language-conditioned VLA training and evaluation. The suite expands from 32 tasks and 4 memory types to 90 tasks spanning 10 types, with natural-language instructions and a shared observation and action interface. It asks two questions: what must a policy remember, and for how long? Task families vary the content and structure of memory, including object attributes, sequences, counts, and pending actions. Difficulty variants adjust factors such as the number of items, sequence length, and delay before recall. Configured episode horizons range from 25 to 2,160 steps, organized into Short, Medium, and Long splits, and retention intervals are characterized separately from episode length. Dense rewards support online RL fine-tuning, and demonstrations for every task support imitation learning. A common evaluation protocol fixes the policy interface, episode seeds, and success metric. Our main contributions are as follows: •A broad, parameterized benchmark for memory in VLA policies. We provide 90 language-conditioned manipulation tasks across 10 memory types, with difficulty variants spanning memory content and temporal demands, and three horizon splits covering 25–2,160 steps (Section 3). •Demonstrations and rewards for policy training. We release 22,500 oracle demonstra- tion episodes, comprising 6,306,953 timesteps, collected using PPO policies and motion planning. Every task is available in both RLDS and LeRobotDataset v3 formats, with dense environment rewards supporting subsequent RL fine-tuning (Section 4). •Standardized evaluation and a reference analysis. We provide a common evaluation protocol in the released code, a per-task characterization of retention intervals, and a reference π 0.5 baseline on 14 tasks (Section 5). 2 Related Work Memory in reinforcement learning. RL memory evaluation includes both diagnostic tasks and diverse partially observable environments. T-Maze and bsuite probe retention across delays [Bakker, 2001, Osband et al., 2020], while POPGym and Memory Gym broaden the range of observations, task structures, and temporal demands [Morad et al., 2023, Pleines et al., 2025]. Other work separates memory from credit assignment [Ni et al., 2023] or develops taxonomies of memory capabilities [Cherepanov et al., 2025]. MIKASA-Robo applies this diagnostic framing to robotic manipulation [Cherepanov et al., 2026a]. We extend its RL setting to language-conditioned policies, with an expanded taxonomy, a common VLA interface, demonstrations, and dense rewards. 2 Memory architectures for robot policies.Generalist policies typically condition actions on limited visual history: OpenVLA uses a single image [Kim et al., 2024], while RT-1, Octo, and RoboVLMs use windows of six, two, and 16 frames, respectively, in their reported main configurations [Brohan et al., 2023, Octo Model Team et al., 2024, Li et al., 2026]. Explicit memory extends this design space through perceptual and cognitive storage in MemoryVLA [Shi et al., 2026], recurrent state inμVLA [Cherepanov et al., 2026b], experience retrieval in MemER [Sridhar et al., 2025], and visual–textual memory at multiple scales in MEM [Torne et al., 2026]. MIKASA-Robo-VLA provides tasks for comparing such designs across memory contents and retention intervals. Its information-gap analysis relates task structure to observation windows without treating window size as a limit on recurrent or external memory. General-purpose manipulation benchmarks. CALVIN evaluates language-conditioned skill composition [Mees et al., 2022], LIBERO studies lifelong knowledge transfer [Liu et al., 2023], SIMPLER targets correspondence between simulated and real-world policy performance [Li et al., 2024], and RoboCasa broadens evaluation to everyday kitchen activities [Nasiriany et al., 2024]. Their standard evaluations do not systematically vary memory content and retention delay. A long sequence can remain locally observable, while a short task can depend on a cue that has disappeared. MIKASA-Robo-VLA makes this distinction explicit. Memory-dependent manipulation benchmarks.MemoryBench introduces spatial-memory and action-recall tasks [Fang et al., 2025]. LIBERO-Mem formalizes four object-centric non-Markov factors [Chung et al., 2026], and RMBench distinguishes one-shot from repeated-exploration mem- ory [Chen et al., 2026]. RoboMME covers temporal, spatial, object, and procedural memory across sixteen tasks [Dai et al., 2026], while RoboMemArena composes longer tasks and annotates memory- relevant keyframes [Lei et al., 2026]. RoboTwin-MeM isolates transient intermediate visual evidence and explicitly scales the number of required keyframes [Yang et al., 2026], and MEMOBench in- stead diagnoses storage, update, and compression through executable checkpoints [Sun et al., 2026]. MIKASA-Robo-VLA adds 90 parameterized tasks over 10 memory types (Table 1), dense rewards, a common evaluation protocol, and 22,500 demonstrations in RLDS and LeRobotDataset v3 – the only suite among these to combine controlled retention and load variation, horizon stratification, dense rewards, and GPU-parallel execution (Table 35, Appendix I). 3 MIKASA-Robo-VLA Benchmark MIKASA-Robo-VLA comprises 90 language-conditioned manipulation tasks implemented in Man- iSkill 3 [Tao et al., 2025]. The benchmark organizes evaluation around the information a policy must retain and the temporal structure of the interaction. Most tasks present a cue that becomes unavailable before or during the action it determines. 10 tasks keep the relevant information visible and serve as reactive controls. Figure 1 illustrates the cue-to-action structure in two representative tasks. The registered variants belong to 26 task families, which in turn are assigned to 10 memory categories. A typical memory-dependent episode presents a cue, hides or occludes it, and then opens the action phase that depends on it. This is not a universal template: reactive controls retain a visible cue, tracking tasks require following a moving referent, andTimedTransferrequires acting in a specified time window. 3.1 Memory taxonomy and task families The suite covers 10 memory types (Table 1). These categories describe the content and operations involved in memory: retaining an identity or location, remembering a set or ordered sequence, tracking changes, or maintaining pending intentions and completed actions. Within these categories, 26 task families share an underlying environment design. Difficulty variants change factors such as the number of distractors, the number of items to retain, and the length of a demonstrated sequence. For example,RememberColorvaries the candidate set, while its timing variants vary the sampled interval between cue and action. These semantic and temporal changes are useful controls, but the benchmark does not form a fully crossed factorial design. Category assignments follow the released manifest. Section 6 discusses an inconsistency between one family’s label and its success condition. 3 086171257343429514600 Environment Step Language Instruction: Observe the cube's color, wait, then touch the cube of the same color. Memory: Object. Task: RememberColor5-Long. 086171257343429514600 Environment Step Language Instruction: Observe which color is under each cup, track the cups as they shuffle, then touch the cup matching the lamp color. Memory: Tracking. Task: ShellGameShuffleColorLampTouch-Long. Figure 1: Memory-dependent manipulation in MIKASA-Robo-VLA. Each row shows 8 evenly spaced frames from a 600-step rollout horizon, together with the task and language instruction. Ticks locate each frame at its nominal environment step. The relevant cue disappears before the dependent action begins. Images show the top-camera view at render resolution, and policy inputs have resolution 128× 128× 3. Appendix K provides rollouts for all tasks. 3.2 Temporal structure and horizon splits Configured episode horizons range from 25 to 2,160 steps. The Short, Medium, and Long splits group tasks by these limits (Table 2), providing a common stratification for training and evaluation. Several families include longer-horizon variants that extend the interval between observing a cue and acting on it while preserving the underlying task semantics. Semantic and temporal parameters therefore offer distinct ways to vary difficulty, although the release is not a fully crossed experimental design. Episode horizon also includes observation, waiting, and motor execution, so it is not itself a measure of retention (Section 5.1). 3.3 Extension over MIKASA-Robo MIKASA-Robo-VLA extends MIKASA-Robo from a representative VLA subset to a full-suite language interface. It broadens task and memory coverage, adds temporal variants, standardizes policy evaluation, and releases demonstrations in both RLDS and LeRobotDataset v3. The two suites share a memory-first design, but task-name overlap alone is not evidence that every variant has an identical environment implementation. Task parameters are meant to be re-specified: Appendix J.3 demonstrates this with custom variants built to test a different memory update rate. 3.4 Observations, actions, and language All tasks expose a shared policy interface. Visual observations consist of a static top-down RGB image and a wrist-camera RGB image, each of resolution 128× 128× 3. A 7-dimensional proprioceptive vector contains end-effector position, roll–pitch–yaw orientation, and gripper state. Actions comprise 7 dimensions for delta end-effector pose and gripper control, executed usingpd_e_delta_poseat 10 Hz. Language instructions specify the goal and the relevant memory operation.For example, RememberColorasks the robot to observe a cube’s color and later select a matching cube. Dif- ficulty variants change the distractor set while retaining the same instruction. The suite uses 30 instruction templates across 90 tasks. Most are shared within families.TimedTransferincludes the required step count, and the Rotate families insert the episode’s target angle. 3.5 Environment interface and evaluation The shared interface standardizes camera and proprioceptive observations and, where required by the collection curriculum, freezes actions during cue presentation or waiting phases. Optional diagnostic overlays affect rendered inspection videos rather than policy images, and the released demonstrations 4 Table 1: Memory-type taxonomy: definitions, task and episode counts, and the realized episode- length distribution in steps (minimum, mean, median, maximum). Figure 1 shows a rollout from two of these types. Appendix K shows one for every task. Each swatch is that type’s color in every figure of this paper. Memory type DefinitionTasks EpisodesMin tMean tMedian tMax t ObjectThe cue reveals which object identity (color or shape) is the target. The agent must retain that identity while acting among visually similar distractors. 184,5001017853599 SpatialThe goal is a place rather than an object identity: which con- tainer hides the ball, where a moving ball will arrive, a target angle, or where an object began. Where that place is hidden, the agent must recall it rather than read it off the current frame. 10 of the 14 are reactive controls whose cue is never hidden, so recall is demanded only in the other 4. 143,50010302890 CapacityThe cue presents a set of items to be collected or touched in any order. The agent must retain the full set membership across the interaction, with set size scaling the memory load. 123,000121431355 1199 TemporalThe cue is a countable or timed event (e.g., a number of blinks). The agent must retain a count or duration and reproduce it during the action phase. 123,00057338204 1004 NegativeThe cue presents a set of items and the goal is defined by exclu- sion. The agent must retain the whole set it was shown in order to touch the one later item whose color or shape was not in it. 92,25010151528 SequentialThe cue presents an ordered sequence of items. The agent must retain both the set and the order and reproduce that order during the action phase. 61,500131489371 1199 ProceduralThe cue demonstrates a motion or shape being traced. The agent must retain a trajectory rather than a static fact and reproduce it during the action phase. 61,500188460307 1480 ProspectiveA brief cue appears part-way through a longer manipulation task and specifies an action to perform only once that task is complete. The agent must hold the pending intention while the manipulation continues to occupy it. 51,250104513514940 TrackingThe cue’s referents are shuffled after the cue ends. The agent must keep updating where the cued content now is rather than retaining a single static fact. 41,00029188102513 ChecklistThe task requires testing several items in an unspecified order. The agent must retain both which items it has already tested and how each test came out in order to report the result set without repeats or omissions. 41,000351805720 1523 Table 2: Horizon-split composition: task and episode counts, frame totals, realized episode length, and the design horizon range. SplitTasks EpisodesFrames Median length (steps) Max length (steps) Horizon range Short389,500303,52718197 25–200 Medium307,500 2,154,062272599 250–600 Long225,500 3,849,3646981,523 700–2,160 were collected with overlays disabled. The exact factory call and wrapper behavior are documented in Appendix E. The benchmark supplies a common evaluation procedure with fixed episode seeds, a 50-episode- per-task default, and success credited if the task condition holds at any step (success_once). The reference baseline uses a smaller budget (Section 5.2), and known interface issues are summarized in Section 6. 3.6 Rewards and success criteria Every task provides a dense reward for RL training and an explicit binary success condition for evaluation. Reward shaping combines reaching and placement terms with task-specific progress signals, such as the number of correctly recalled items, and penalties for unnecessary motion or disturbing protected objects. Terminal success receives a fixed reward. Success is assessed directly from task conditions — for example, reaching a goal region, matching a count, or completing a 5 sequence in order — rather than by applying a threshold to the shaped reward. Appendix F specifies the reward components and success conditions for each family. 4 Datasets We release demonstrations for every benchmark task to support supervised VLA fine-tuning and subsequent evaluation. The collection contains 22,500 episodes and 6,306,953 timesteps, correspond- ing to 175.2 hours of simulated interaction at10 Hz. Both release formats use the policy interface specified in Section 3.4. 4.1 Demonstration collection Each task contributes 250 successful oracle episodes. Demonstrations are generated by PPO policies trained with dense rewards for 34 tasks and by scripted motion planners for the remaining 56. Both use privileged simulator state to resolve the task, and initial scenes are sampled through the environment’s reset procedure. Simulator seeds are retained in the RLDS metadata for reproducing initial conditions. The collected rollouts are rendered through the VLA observation wrappers to produce camera images and proprioception in the same representation used for policy evaluation. Only episodes satisfying the task’s success condition are retained. Consequently, the reported oracle success rate of 1.000 describes the filtered dataset, rather than an unconditional estimate of oracle reliability. The release provides successful behavior for imitation learning but does not systematically cover failure recovery or states induced by a learned policy’s errors. 4.2 Release formats The same demonstrations are available in RLDS [Ramos et al., 2021] and LeRobotDataset v3 [Cadène et al., 2026]. Both contain language instructions, two RGB camera streams, proprioception, and actions. The RLDS release follows the TFDS organization used by Open X-Embodiment [Open X-Embodiment Collaboration, 2024], stores camera frames as PNG images, and retains per-step rewards. The LeRobot release stores state and action arrays in Parquet and camera streams as AV1 video. It occupies 25.9 GiB, compared with 287.4 GiB for RLDS. The releases differ in both image encoding and schema, so their size difference should not be attributed to compression alone. The formats differ in metadata availability: the released LeRobot schema omits per-step rewards, so reward-dependent training should use RLDS. No shared episode identifier is provided, so episode indices must be treated as local to each format, and seeds, rewards, and success metadata should be read from the corresponding RLDS records rather than joined by index. Appendix A documents the schemas and metadata limitations. 4.3 Dataset composition The fixed number of demonstrations per task balances episode counts, but longer tasks contribute more timesteps. Figure 2 shows the realized episode-length distribution by horizon split and memory type. The horizon splits describe task duration, rather than prescribing a single train/test partition, and evaluation must separately specify held-out seeds or tasks. Table 2 summarizes the split-level composition, and Appendix G reports per-task statistics. 5 Analysis We first characterize environment-level information gaps, then report a reference policy experiment, and finally examine associations within its limited evaluated subset. 5.1 Information gap and observation context The information gap measures the interval between the last availability of a task-relevant cue and the start of the phase that requires a cue-dependent action. We extract this interval from the environment’s cue and blackout phase boundaries, rather than from a policy’s realized action time. Fixed phase timings yield a constant gap, while timings sampled at reset yield a distribution, summarized by its 6 101001000 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of episodes ≤ length (a) Length CDF by horizon split Short Medium Long Pooled 101001000 Negative Spatial Object Tracking Temporal Procedural Capacity Sequential Prospective Checklist (b) Length by memory type, ordered by median Episode length (steps, log scale) Figure 2: Realized episode lengths across 22,500 demonstrations. (a) Empirical cumulative distributions for the full release and each horizon split, with a logarithmic horizontal axis. (b) Length distributions by memory type, ordered by median. Violins show kernel-density estimates. Horizontal lines indicate the observed range and ticks mark the median. median and interquartile range. Related formulations characterize memory length or complexity through optimal policies [Ni et al., 2023, Chen et al., 2026]. Our measure provides an environment- based description without requiring an optimal policy estimate. Per-family descriptions in Appendix F specify the underlying task structure. An environment-determined gap is available for 70 tasks. The remaining tasks comprise 10 reactive controls, whose cue remains visible, and 10 tasks whose retention interval depends on reaching a state and cannot be determined from phase timing alone. Among the determined gaps, 28 exceed 16 steps, the widest fixed observation window reported for the configurations reviewed in Section 2, counted in successive observations at the benchmark’s control rate. This threshold is a reference point, not a statement that all VLA policies use the same context length, and it does not imply that the tasks below it are memory-free – the gap counts only the provably-cue-absent interval, and every memory-dependent task in the suite requires retention by construction. The gap measures an explicit interval of information unavailability, rather than the total duration for which a successful policy must retain it. For 15 tasks, the gap is zero because the dependent action phase begins immediately when the cue disappears, and recall may still be needed throughout that phase. The threshold counts therefore omit some memory demands, including those in Spatial and Procedural tasks with zero determined gaps and Prospective and Checklist tasks without deter- mined gaps. Conversely, a positive gap alone does not establish that a policy must fail: correlated observations, embodied strategies, or explicit memory may provide alternative access to the relevant information. 5.2 Reference baseline Training and evaluation.We fully fine-tuneπ 0.5 [Physical Intelligence et al., 2025] from the public pi05_basecheckpoint on 14 tasks, using 3,500 LeRobot episodes comprising 798,296 timesteps. The subset spans 7 memory types and 12.7% of the released timesteps. It follows the available training mixture and is not a representative sample of the full benchmark. Temporal, Negative, and Checklist are not evaluated. The policy receives the current images from both cameras, resized to224× 224, and proprioception discretized into the text prompt. It has no observation history or explicit memory module. The 7 active action dimensions are padded to the checkpoint’s 32 dimensions, and predictions are executed in open-loop chunks of 8 actions. We evaluate one training run at checkpoint 139,999, using 20 episodes per task with seeds disjoint from the training set. Success followssuccess_once. This evaluation uses fewer episodes and a different seed range than the released defaults, with the wrapper adjustments of Section 6. Appendix H reports the full configuration. 7 Table 3: Referenceπ 0.5 performance with current images and proprioception but no observation history or explicit memory module, evaluated on 20 episodes per task. The aggregate is the unweighted mean across tasks. TaskMemory typeHorizonSuccess BunchOfColors3-Long-VLA-v0CapacityLong0.000 ChainOfColors3-Long-VLA-v0SequentialLong0.000 GatherAndRecall1-VLA-v0ProspectiveShort0.350 GatherAndRecall3-VLA-v0ProspectiveMedium0.150 InterceptGrabMedium-VLA-v0SpatialShort0.050 InterceptMedium-VLA-v0SpatialShort0.400 RememberColor5-Long-VLA-v0ObjectMedium0.300 RememberColor5-VLA-v0ObjectShort0.250 RememberShape5-VLA-v0ObjectShort0.050 RememberShapeAndColor3x2-VLA-v0ObjectShort0.200 ShellGameShuffleColorLampTouch-Long-VLA-v0TrackingMedium0.350 TakeItBack-VLA-v0SpatialShort0.500 TraceShapeEasy-VLA-v0ProceduralMedium0.300 TraceShapeSeqEasy-VLA-v0ProceduralLong0.050 Mean (14 tasks)0.211 Overall performance and chance references.Mean task success is0.211± 0.044(Table 3), with the aggregate standard error computed across tasks. Per-task binomial standard errors range from 0.049 to 0.112 for nonzero success rates. Zero-success rows instead report exact one-sided 95% upper bounds. The small evaluation budget limits the precision of individual task comparisons. Uniformly random actions achieve 1 success in 4,500 episodes across all 90 tasks. This establishes a low floor for unstructured control, but does not exclude success through competent manipulation combined with guessing. For the 7 evaluated tasks with a discrete choice amongkequally likely candidates, a cue-independent uniform choice has success probability1/k. Across these tasks, the baseline obtains 33 successes in 140 episodes, compared with 35.3 expected successes and a standard deviation of 5.1 under this reference. No task exceeds its choice reference by more than one reported standard error. These results provide no clear evidence of cue-dependent selection on this subset. The analytic reference assumes successful execution after a candidate is selected and is not a learned-policy baseline. 5.3 Memory architecture comparison μVLA [Cherepanov et al., 2026b], an explicit-memory extension of the benchmark’s task family, reports the comparison this baseline alone cannot: OpenVLA-OFT with and without a 64-token recurrent memory module raises mean success from 0.42 to 0.84 on five trained tasks and from 0.07 to 0.23 on 18 held-out ones [Cherepanov et al., 2026b] (per-task breakdown in Appendix J). A separate execution-regime analysis there shows the same open-loop chunking this baseline uses shifts per-task success by up to 0.650, a genuine confound on the Long-split result below. 5.4 Performance across memory demands and horizons All permutation tests below use tasks as the unit of analysis, enumerate all assignments with the observed group sizes, and describe associations within the evaluated subset. Information gaps. The 9 tasks with determined gaps have mean success 0.167, compared with 0.290 for the other 5 tasks (two-sided permutationp = 0.209). The latter group includes 2 reactive controls and 3 tasks with undetermined retention intervals, so this comparison does not contrast memory-dependent and memory-free tasks. Excluding the 3 Long-split tasks reduces the difference from 0.123 to 0.048 (p = 0.617). Among tasks with determined gaps, Spearman’s correlation between median gap and success isρ = 0.161(two-sided permutationp = 0.673). Neither comparison establishes an association at this sample size, and the small subset also limits the ability to detect one. 8 Episode horizons. The 3 evaluated Long-split tasks achieve 1 success in 60 episodes (0.017), compared with 58 in 220 (0.264) for the remaining 11 tasks. A one-sided task-level permutation test gives p = 0.008. Split membership follows Table 2’s horizon bands, not the -Long- name suffix. This contrast identifies a concentration of failures in the longest-horizon tasks, but its cause is unresolved: those tasks include the subset’s only Capacity and Sequential tasks and one Procedural task, coupling duration with the need to reproduce sets, orders, or trajectories, and Section 5.3 above already documents open-loop chunking as an independent confound on longer rollouts. 6 Limitations and Future Work The reference results cover one training run and one checkpoint on 14 of 90 tasks, and a separate architecture comparison from a related project, not a matched run, does not show which tasks discriminate under identical training. The baseline’s 8-action open-loop chunks and training on successful demonstrations with limited recovery mean a long-task failure can reflect memory, motor error, or distribution shift. Phase timing captures explicit information gaps, not all policy-dependent retention demands, and does not rule out shortcuts. The released gaps are not a dense delay sweep, and TimedTransfermeasures elapsed-time tracking, not recall of a hidden visual cue. All environments and demonstrations are simulated, collected with privileged-state PPO policies or motion planners, so transfer to physical robots is unmeasured and the release contains no human demonstrations. The instruction set is limited to 80 tasks sharing an instruction and no held-out paraphrases, so results concern task execution, not linguistic generalization. 7 Conclusion MIKASA-Robo-VLA extends MIKASA-Robo from a reinforcement-learning suite into a language- conditioned benchmark for memory in vision-language-action policies. Every one of its 90 tasks carries an instruction, spans 10 memory types, and falls into one of 3 horizon splits, with step horizons from 25 to 2,160. We release 22,500 oracle trajectories, 6,306,953 frames total, in both RLDS and LeRobotDataset v3, and for 28 tasks the environment’s own phase timings place the median information gap beyond the widest fixed-context window we survey. Every memory-dependent task in the suite requires retention by construction, regardless of the measured gap length. A reference π 0.5 baseline with current images and proprioception but no observation history or explicit memory module reaches0.211± 0.044mean success on 14 tasks. That number describes one model under one training and evaluation protocol, not a ceiling on what memory-free policies can do on this suite, and several confounds keep it from separating horizon length from retention demand. The benchmark’s contribution is the standardized task suite, the dataset, and the diagnostic apparatus around it (the taxonomy, the gap metric, the chance floors) rather than this single evaluated point. We see MIKASA-Robo-VLA as infrastructure for the question its title asks, not an answer to it. Its 26 task families and released oracle trajectories give later work a fixed target for training and comparing memory-augmented VLA policies against a documented baseline, with the same per-family reward and success definitions available to every method that evaluates against it. References Junjie Bai, Yu-Wei Chao, Qizhi Chen, Jinwei Gu, Moo Jin Kim, Zhaoshuo Li, Xuan Li, Tsung-Yi Lin, Ming-Yu Liu, Nic Ma, Kaichun Mo, Delin Qu, Shangkun Sun, Hongchi Xia, Fangyin Wei, and Xiaohui Zeng. Openpi comet: Competition solution for 2025 behavior challenge. arXiv preprint, 2025. Bram Bakker. Reinforcement learning with long short-term memory. In Advances in Neural Information Processing Systems (NeurIPS), volume 14, 2001. Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil J. Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Deeksha Man- junath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Kevin Sayed, 9 Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Vanhoucke, Steve Vega, Quan Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Tianhe Zhang. RT-1: Robotics transformer for real-world control at scale. In Robotics: Science and Systems (RSS), 2023. Rémi Cadène, Simon Aliberts, Francesco Capuano, et al. LeRobot: An open-source library for end-to-end robot learning. In International Conference on Learning Representations (ICLR), 2026. Tianxing Chen, Yuran Wang, Mingleyang Li, Yan Qin, Hao Shi, Zixuan Li, Yifan Hu, Yingsheng Zhang, Kaixuan Wang, Yue Chen, Hongcheng Wang, Junjie Wang, Tianhang Yang, Renjing Xu, Ruihai Wu, Yao Mu, Yaodong Yang, Hao Dong, and Ping Luo. RMBench: Memory-dependent robotic manipulation benchmark with insights into policy design. arXiv preprint, 2026. Egor Cherepanov, Nikita Kachaev, Artem Zholus, Alexey K. Kovalev, and Aleksandr I. Panov. Unraveling the complexity of memory in RL agents: an approach for classification and evaluation. In Conference on Robot Learning (CoRL), 2025. Egor Cherepanov, Nikita Kachaev, Alexey K. Kovalev, and Aleksandr I. Panov. Memory, benchmark & robots: A benchmark for solving complex tasks with reinforcement learning. In International Conference on Learning Representations (ICLR), 2026a. Egor Cherepanov, Nikita Kachaev, Daniil Zelezetsky, Aydar Bulatov, Artem Pshenitsyn, Yuri Kuratov, Alexey Skrynnik, Aleksandr I. Panov, and Alexey K. Kovalev.μVLA: On recurrent memory for partially observable manipulation in VLA models. arXiv preprint, 2026b. Nhat Chung, Taisei Hanyu, Toan Nguyen, Huy Le, Frederick Bumgarner, Duy Minh Ho Nguyen, Khoa Vo, Kashu Yamazaki, Chase Rainwater, Tung Kieu, Anh Nguyen, and Ngan Le. Rethinking progression of memory state in robotic manipulation: An object-centric perspective. In AAAI Conference on Artificial Intelligence (AAAI), 2026. Yinpei Dai, Hongze Fu, Jayjun Lee, Yuejiang Liu, Haoran Zhang, Jianing Yang, Chelsea Finn, Nima Fazeli, and Joyce Chai. RoboMME: Benchmarking and understanding memory for robotic generalist policies. In International Conference on Machine Learning (ICML), 2026. Haoquan Fang, Markus Grotz, Wilbert Pumacay, Yi Ru Wang, Dieter Fox, Ranjay Krishna, and Jiafei Duan. SAM2Act: Integrating visual foundation model with a memory architecture for robotic manipulation. In International Conference on Machine Learning (ICML), 2025. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé I, and Kate Crawford. Datasheets for datasets. Communications of the ACM, 64(12): 86–92, 2021. Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. OpenVLA: An open-source vision-language-action model. In Conference on Robot Learning (CoRL), pages 2679–2713, 2024. Huashuo Lei, Wenxuan Song, Huarui Zhang, Jieyuan Pei, Jiayi Chen, Haodong Yan, Han Zhao, Pengxiang Ding, Zhipeng Zhang, Lida Huang, Donglin Wang, Yan Wang, and Haoang Li. RoboMemArena: A comprehensive and challenging robotic memory benchmark. arXiv preprint, 2026. Xinghang Li, Peiyan Li, Long Qian, Minghuan Liu, Dong Wang, Jirong Liu, Bingyi Kang, Xiao Ma, Xinlong Wang, Di Guo, Tao Kong, Hanbo Zhang, and Huaping Liu. Towards generalist robot policies: What matters in building vision-language-action models for generalist robots. Nature Machine Intelligence, 8(2):158–172, 2026. Xuanlin Li, Kyle Hsu, Jiayuan Gu, Karl Pertsch, Oier Mees, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sieh, Sean Kirmani, Sergey Levine, Jiajun Su, Chelsea Finn, and Ted Xiao. Evaluating real-world robot manipulation policies in simulation. In Conference on Robot Learning (CoRL), 2024. 10 Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. LIBERO: Benchmarking knowledge transfer for lifelong robot learning. In Advances in Neural Information Processing Systems (NeurIPS), 2023. Oier Mees, Lukas Hermann, Erick Rosete-Beas, and Wolfram Burgard. CALVIN: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters, 2022. Steven Morad, Ryan Kortvelesy, Matteo Bettini, Stephan Liwicki, and Amanda Prorok. POP- Gym: Benchmarking partially observable reinforcement learning. In International Conference on Learning Representations (ICLR), 2023. Soroush Nasiriany, Abhiram Maddukuri, Lance Zhang, Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Mandlekar, and Yuke Zhu. RoboCasa: Large-scale simulation of everyday tasks for generalist robots. In Robotics: Science and Systems (RSS), 2024. Tianwei Ni, Michel Ma, Benjamin Eysenbach, and Pierre-Luc Bacon. When do transformers shine in RL? decoupling memory from credit assignment. In Advances in Neural Information Processing Systems (NeurIPS), 2023. Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, Jianlan Luo, You Liang Tan, Lawrence Yunliang Chen, Pannag Sanketi, Quan Vuong, Ted Xiao, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open-source generalist robot policy. In Robotics: Science and Systems (RSS), 2024. Open X-Embodiment Collaboration. Open X-Embodiment: Robotic learning datasets and RT-X models. In IEEE International Conference on Robotics and Automation (ICRA), 2024. Ian Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csába Szépesvári, Satinder Singh, Benjamin Van Roy, Richard Sutton, David Silver, and Hado van Hasselt. Behaviour suite for reinforcement learning. In International Conference on Learning Representations (ICLR), 2020. Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Allen Z. Ren, Lucy Xiaoyang Shi, Laura Smith, Jost Tobias Springenberg, Kyle Stachowicz, James Tanner, Quan Vuong, Homer Walke, Anna Walling, Haohuan Wang, Lili Yu, and Ury Zhilinsky.π 0.5 : a vision-language-action model with open-world generalization. In Conference on Robot Learning (CoRL), pages 17–40, 2025. Marco Pleines, Matthias Pallasch, Frank Zimmer, and Mike Preuss. Memory Gym: Towards endless tasks to benchmark memory capabilities of agents. Journal of Machine Learning Research, 26(6): 1–40, 2025. Sabela Ramos, Sertan Girgin, Léonard Hussenot, Damien Vincent, Hanna Yakubovich, Daniel Toyama, Anita Gergely, Piotr Stanczyk, Raphael Marinier, Jeremiah Harmsen, Olivier Pietquin, and Nikola Momchev. RLDS: an ecosystem to generate, share and use datasets in reinforcement learning. arXiv preprint, 2021. Hao Shi, Bin Xie, Yingfei Liu, Lin Sun, Fengrong Liu, Tiancai Wang, Erjin Zhou, Haoqiang Fan, Xiangyu Zhang, and Gao Huang. MemoryVLA: Perceptual-cognitive memory in vision-language- action models for robotic manipulation. In International Conference on Learning Representations (ICLR), 2026. Ajay Sridhar, Jennifer Pan, Satvik Sharma, and Chelsea Finn. MemER: Scaling up memory for robot control via experience retrieval. arXiv preprint arXiv:2510.20328, 2025. Haiyang Sun, Haoxiao Wang, Junming Chen, Weicheng Fang, Zihao Su, Jingkun Yi, Wenyou Yi, Hao Chen, and Zhou Zhao. MEMOBench: A process level memory benchmark for robotic manipulation. arXiv preprint, 2026. 11 Stone Tao, Fanbo Xiang, Arth Shukla, Yuzhe Qin, Xander Hinrichsen, Xiaodi Yuan, Chen Bao, Xinsong Lin, Yulin Liu, Tse-kai Chan, Yuan Gao, Xuanlin Li, Tongzhou Mu, Nan Xiao, Arnav Gurha, Viswesh Nagaswamy Rajesh, Yong Woo Choi, Yen-Ru Chen, Zhiao Huang, Roberto Calandra, Rui Chen, Shan Luo, and Hao Su. Demonstrating GPU parallelized robot simulation and rendering for generalizable embodied AI with ManiSkill3. In Robotics: Science and Systems (RSS), 2025. Marcel Torne, Karl Pertsch, Homer Walke, Kyle Vedder, Suraj Nair, Brian Ichter, Allen Z. Ren, Haohuan Wang, Jiaming Tang, Kyle Stachowicz, Karan Dhabalia, Michael Equi, Quan Vuong, Jost Tobias Springenberg, Sergey Levine, Chelsea Finn, and Danny Driess. MEM: Multi-scale embodied memory for vision language action models. arXiv preprint arXiv:2603.03596, 2026. Ganlin Yang, Zhangzheng Tu, Yuqiang Yang, Sitong Mao, Junyi Dong, Tianxing Chen, Jiaqi Peng, Jing Xiong, Jiafei Cao, Jifeng Dai, Wengang Zhou, Yao Mu, and Tai Wang. EventVLA: Event- driven visual evidence memory for long-horizon vision-language-action policies. arXiv preprint, 2026. Appendix Overview • Appendix A (p. 12): Datasheet • Appendix B (p. 14): Dataset Schemas • Appendix C (p. 16): Observation and Action Details • Appendix D (p. 17): Evaluation Protocol and Release Issues • Appendix E (p. 21): Wrapper Cookbook • Appendix F (p. 22): Per-Family Task Details • Appendix G (p. 43): Full Per-Task Statistics • Appendix H (p. 48): Reference Baseline: Full Configuration • Appendix I (p. 48): Comparison to Existing Benchmarks • Appendix J (p. 48): Supplementary External Evaluations • Appendix K (p. 50): Rollout Contact Sheet A Datasheet Following the framework of Gebru et al. [2021]. Motivation For what purpose was the dataset created? This dataset provides training data and environments for evaluating language-conditioned manipulation tasks with documented memory demands. It supports comparisons of policies that use recent observations, recurrent state, or external memory across task semantics and retention intervals. Composition What do the instances represent? Each episode is one simulated manipulation rollout: two RGB camera streams, a 7-dimensional proprioceptive state, and a 7-dimensional action per step, plus the language instruction for that task (Section 3.4). Do the released frames contain rendered debug text? No. Overlay wrappers modify the output ofrender, not the observation returned bystep, and collection usedinclude_overlays=False. Debug text appears only in the rendered rollout videos used for this paper’s filmstrip figures (Ap- pendix F). How many instances are there? 90 tasks, 250 episodes each, 22,500 episodes and 6,306,953 frames in total (175.2 hours of control at 10 Hz). Does the dataset contain all possible instances, or a sample? Every task’s release is a fixed sample of 250 oracle rollouts, not an exhaustive enumeration of the task’s state space – see Collection Process below for how each episode’s initial state is drawn. 12 Is there a label or target associated with each instance? Each RLDS step stores a normalized dense reward, and the RLDS sidecar stores per-episodesuccess_once, reward sums, and simulator seeds. LeRobot’s native schema stores neither reward nor success. Every released episode satisfies success_once. Appendix B describes the format-specific fields and their alignment. Are there recommended data splits? The benchmark’s own Short/ Medium/Long horizon splits (Table 2) and its 10 memory types (Table 1) are the natural stratification for held-out evaluation subsets. This release does not impose a single train/test split, since different consumers will want to hold out different subsets of the 90 tasks. Does the dataset relate to people? No. Every frame is rendered from a simulated scene holding a robot arm and rigid props, and the demonstrations were generated by policies and motion planners rather than by human teleoperation (Section 4.1), so the release contains no human subjects, no recorded operators, and no personally identifiable information. Collection Process How was the data associated with each instance acquired? Every episode is a full simulator rollout, not a derived or inferred value. On 34 tasks a PPO policy trained against the task’s dense reward is the oracle, and the remaining 56 use a scripted motion planner, for tasks whose action sequence is fully determined once the cue is resolved (Section 4.1). What mechanisms were used to collect the data? Both oracle types run inside the same Man- iSkill/Gymnasium simulator the released observation and action spaces are defined for. Video and proprioception are re-rendered from each rollout after collection using the VLA-facing observation wrapper stack (Section 3.5), so the released data matches exactly what a policy would receive at evaluation time, not the oracle’s privileged view during collection. Over what timeframe was the data collected? The collection dates are not recorded in the release metadata. Each released episode retains a simulator seed inepisode_seeds. Recreating an initial state also requires matching simulator software, assets, and environment configuration. The PPO checkpoints used for 34 tasks are not part of the release, so those rollouts cannot be replayed from the published data alone. Warning: the two release branches are ordered differently, and episode index is not a join key. The RLDS and LeRobot branches contain the same episodes for every task, but their order can differ. The converter does not record a cross-format episode mapping. Episode length cannot supply an unambiguous mapping: all 90 tasks contain at least two episodes with the same length. Thismattersbecauseofwheretheper-episodemetadatalives.Eachtask’s source_rlds_metadata.json– carrying that task’s episode lengths,success_onceflags, reward sums, and simulator seeds – sits inside the LeRobot tree, but its arrays are indexed in RLDS order. The LeRobot branch’s ownmeta/info.jsoncarries no reward, no success flag, and no seed, so that file is the only source for them. A reader who trains on the LeRobot branch and joins this metadata by episode index can assign a different episode’s reward, success flag, or seed. The release provides no unambiguous cross-format join key. The simulator seed appears only in source_rlds_metadata.json, in RLDS order. Read an episode’s reward, success flag, or seed from the RLDS branch, where index and metadata agree, and treat the LeRobot episode index as branch-local. This limitation does not affect any number reported in this paper: every statistic here is a count, sum, or median computed fromsource_rlds_metadata.jsonalone, whose own arrays are mutually aligned, and no analysis in this paper joins across the two branches. Preprocessing, Cleaning, and Labeling Was any preprocessing done? Oracle rollouts are filtered to episodes that reachsuccess_once. Both release formats apply the same observation wrapper stack (Section 3.5) before writing video and proprioception, so raw simulator state is never released directly. No other filtering, augmentation, or relabeling is applied. Is the raw data also available? No. Only the wrapped, VLA-facing observation is released. The underlying ManiSkill/SAPIEN scene state is not retained per episode. 13 Uses What tasks has the dataset been used for? Training and evaluating the referenceπ 0.5 baseline reported in Section 5.2 of this paper, on a 14-task subset. Is there anything about the composition or collection that might impact future uses? The oracle demonstrations come from PPO policies and motion planners with privileged simulator access, not human teleoperation (Section 6). They are a controlled source of training data for exactly these tasks, not a proxy for the demonstration distributions a real deployment would collect. Every episode succeeds by construction (Section 4.1), so the release contains no failure trajectories for a policy to learn to avoid. Are there tasks for which the dataset should not be used? It should not be used to argue that a policy is ready for hardware. Every episode is simulated and every one succeeds, so the release holds no failure and no recovery for a policy to learn from, and no number in this paper was measured against a physical robot. A score here is also not by itself evidence of retention: Section 5.1 measures what a task demands, not what a policy does. Distribution Will the dataset be distributed to third parties?It already is, in both formats (RLDS: https://huggingface.co/datasets/mikasa-robo/mikasa-robo-vla-rlds;LeR- obot:https://huggingface.co/datasets/mikasa-robo/mikasa-robo-vla-lerobot), alongside the environment suite and the wrapper stack that produced it (https: //github.com/CognitiveAISystems/MIKASA-Robo). Both predate this submission. Under what license? The release distributes the benchmark code and trajectories under the MIT License. Users should consult the licenses of upstream software and assets separately. Maintenance Will the dataset be updated? Corrections to the manifest or extraction scripts, if found after release, will be documented in the repository’s changelog rather than silently changing released files. B Dataset Schemas Section 4.2 names the two release formats and what distinguishes them. This section gives their per-key schemas, so that a loader can be written against this release rather than against RLDS and LeRobot in general. Every literal below is quoted from a named source file or from the released schema itself. None is a measurement of the data. Actions are normalized, but proprioception is not. The control mode ispd_e_delta_pose (Section 3.4), which resolves to ManiSkill’sPDEEPoseControllerConfigfor the arm and PDJointPosMimicControllerConfigfor the gripper. Both setnormalize_action=Trueby default, and the Panda’s registered configuration does not override it, so the environment’s action space is[−1, 1] 7 and the controller rescales into physical units internally. The stored actions are the normalized ones. Reading channel by channel, against the bounds the Panda configuration sets: •Channels 1–3, delta end-effector position:pos_lower=-0.1,pos_upper=0.1. A stored +1 commands 0.1 m of translation on that axis for that step. • Channels 4–6, delta end-effector orientation:rot_lower=-0.1,rot_upper=0.1. These three are not rescaled the way the position channels are. The controller clips them by their joint norm and multiplies byrot_loweralone.rot_uppersets the declared action-space bound and takes no part in the scaling. A stored+1therefore commands−0.1 rad. The magnitude a reader would infer from the position channels is right and the sign is backwards. Replaying stored actions through the same controller is unaffected, since the convention is self-consistent, but converting them to physical units is not. • Channel 7, gripper:lower=-0.01,upper=0.04, anduse_deltaisFalseon this con- troller. Unlike the six above it, this channel is an absolute target finger position, not a delta, 14 and its bounds are not symmetric about zero. A stored0commands the midpoint of that range, not an unchanged gripper. The 7-dimensional proprioceptive vector is stored raw, in the simulator’s own units, and is not normalized or centered:end-effectorxyzin metres, roll-pitch-yaw in radians as ConvertJointsToEEFXyzRpyGripperWrapperderives them from the end-effector quaternion, and a gripper channel holding the sum of the two Panda finger joint positions, also in metres. A loader that rescales state to match action, or rescales action to match state, produces a policy that trains without error and acts on the wrong scale. LeRobotDataset v3 branch. Each task directory holdsmeta/info.json, columnar Parquet underdata/, and one AV1-encoded MP4 per camera undervideos/. The metadata declares codebase_version v3.0androbot_type mikasa_robo. Table 4 lists the feature schema, which is identical across all 90 tasks. Table 4: LeRobotDataset v3 feature schema, from each task’smeta/info.json. Video frames decode to HWC uint8. The declared shape is the frame shape, not the per-row shape. KeyTypeNotes observation.images.topvideo128 × 128 × 3, AV1 inyuv420p,10 Hz, static camera observation.images.wristvideoas above, wrist-mounted camera observation.state float32 [7]physical units, not normalized action float32 [7]normalized to [−1, 1] timestamp float32 [1]seconds from episode start frame_index int64 [1]step index within the episode episode_index int64 [1] branch-local, not a join key across branches (Ap- pendix A) index int64 [1]row index within the task task_index int64 [1]resolves againstmeta/tasks.parquet, where the language instruction lives There is no reward, no success flag, and no simulator seed anywhere in this branch’s own metadata. All three come from thesource_rlds_metadata.jsonfile that sits in the same directory but is indexed in RLDS order, which is the alignment hazard the datasheet documents (Appendix A). RLDS branch.Each task directory holds a single TFDS version directory,1.0.0, with dataset_info.json,features.json, and sharded TFRecords. The top-levelFeaturesDict has two members: astepssequence, whose per-step schema is Table 5, and anepisode_metadata dict whose only field isfile_path, aTextpointing at the source file the episode was converted from. Nothing else is carried per episode. The seed and the success flag are absent from the TFDS record as well. They come from the sidecar this branch writes besidefeatures.jsonasmetadata.json, which is the same file, byte for byte, that the LeRobot branch ships in each task directory under the namesource_rlds_metadata.json. The schema below is read off one task’sfeatures.json. The 90 files share 1 MD5 digest,01a88910206da914198d39bd053a8def, so it describes the whole branch. Whichrewardisstored.Therewardfieldholdseachenvironment’s compute_normalized_dense_reward, not itscompute_dense_reward.Both dataset col- lectors construct the environment withreward_mode="normalized_dense"(Appendix E shows the same call), which routes ManiSkill’sget_rewardto the normalized method, and the RLDS builder writes that per-step value through unchanged. Each environment implements the normalized method as its dense reward divided by its own maximum-reward constant, and that constant differs between families. The stored values are therefore on a common per-step scale across tasks, and are a fixed rescaling of the reward terms the per-family summaries describe (Appendix F). Recovering the unscaled dense reward means multiplying by the environment’s own constant. Two things in the released schema will mislead a reader who trusts the documentation strings over the values. Therewardfield’s own description reads “Reward if provided, 1 on final step for demos”, and is_terminal’s reads “True for demos”. Both are inherited verbatim from the Open X-Embodiment builder template and neither describes what the builder writes. The stored rewards are dense and per-step, as the episode reward sums in the sidecar metadata show. 15 Table 5: RLDS per-step schema, from each task’sfeatures.json. The Notes column gives the corresponding LeRobot key where one exists. Key under stepsTypeNotes observation/image uint8 128× 128× 3PNG-encoded,→ observation.images.top observation/wrist_image uint8 128× 128× 3PNG-encoded,→ observation.images.wrist observation/proprio float32 [7]→ observation.state action float32 [7]normalized to [−1, 1] reward float32no LeRobot counterpart, see below discount float32 written as the constant1.0on every step, so it carries nothing is_first, is_last boolepisode boundaries is_terminal boolthe environment’s done flag language_instruction Text repeated per step,→ task_indexplus meta/tasks.parquet The branches were checked to agree. Section 4.2 states that both branches hold the same trajectories. Each task’s LeRobottotal_frames, written by the converter, equals the sum of that task’s RLDS- orderepisode_lengthson 90 of 90 tasks, for 6,306,953 frames each way. The camera keys, image shape, control rate, and state and action dimensions are the same value on every task. Both checks run in this paper’s test suite rather than having been done once by hand. What they constrain is totals and schema. A per-episode correspondence does not follow from them, and the datasheet’s warning about episode order stands (Appendix A). C Observation and Action Space Details Section 3.4 gives the observation and action space at the level a reader needs to understand the benchmark. This appendix gives the level a reader needs to reproduce it. Observation dictionary. After the wrapper stack (Section 3.5) is applied, every task’s observation is a dictionary with four keys, not the two a policy needs. Two are the intended policy input: the two stacked camera images (top and wrist, each128× 128× 3uint8) and the 7-dimensional end-effector state (position, roll-pitch-yaw orientation, and a scalar gripper state, derived from the robot’s joint state). The other two are diagnostic channels carrying privileged state, one of which was observed holding a real value on all 90 canonical tasks when each was reset and stepped. They are injected by the innermost wrapper in the stack, which is applied unconditionally to every task and cannot be switched off. A task that defines neither gets the same sentinel value in both slots (the numeral 4242424242, coincidentally the same literal the evaluation module uses as its start seed, Appendix D) rather than having the keys omitted. The released episodes carry neither key – the extraction path writes only the two camera images, the proprioceptive state, and the action – so this concerns live evaluation and any pipeline that steps an environment directly. The language instruction is not in the observation. It is returned alongside the reward, success flag, and other per-step information rather than as part of the observation dictionary. Each task carries one sentence, shared with the other variants of its family (Section 3.4) and identical across that task’s episodes except in the two Rotate families, whose instruction names that episode’s target angle. Those two families are also the only ones whose field is a list rather than a string: they format one sentence per parallel environment, where every other task returns the single string its class declares. A loader written against the common case tokenizes a list on them. One written against them indexes the first element and gets a single character everywhere else. Beyond the four observation keys, no other simulator state – object poses, joint torques, contact forces – is exposed. Oracle policies see this information during data collection, but the released episodes are re-rendered from the same rollouts using the VLA-facing wrapper stack, so training data matches exactly what a policy receives at evaluation time. Warning: one diagnostic channel leaks the answer on 6 tasks, and a policy must drop it. The suite’s own reinforcement-learning training wrapper drops both diagnostic keys on its image- 16 observation path, before the observation reaches a policy. The VLA wrapper stack has no equivalent step: the innermost wrapper injects the two keys and nothing downstream removes them (Appendix E). A task that assigns the affected channel a real value assigns it once, at reset, and echoes it unchanged with nothing gating it on elapsed time, so it survives into the observation a policy sees at every step, including the steps after the cue is meant to be hidden. On 10 canonical ids it was read holding a real value at both reset and episode end – the target color, the shape sequence, or the goal angle: • RotateLenientPos ,RotateLenientPosNeg,RotateStrictPos, RotateStrictPosNeg • ShellGameColorLampTouch ,ShellGameShuffleColorLampTouch, ShellGameShuffleColorLampTouch-Long • TraceShapeSeqEasy, TraceShapeSeqMedium, TraceShapeSeqHard (each with the-VLA-v0suffix). The two groups are not the same problem. On the 4Rotate*ids the cue is visible by design: those tasks regenerate the target angle into the language instruction at every step, the angle is therefore already available to the policy through a channel it is supposed to read, and this paper classes all 4 as reactive controls that demand no retention (Section 5.1). Nothing is concealed there for that diagnostic channel to reveal. The instruction rounds the angle to whole degrees and the diagnostic channel does not, so the channel is the more precise of the two. It is not the more useful one: these tasks accept any final angle within0.1 radof the target, an order of magnitude past the half-degree the rounding can cost. On the remaining 6 tasks listed above, the cue is supposed to be hidden after the cue phase, and these tasks are counted in this paper’s retention statistics on that basis. That the diagnostic channel still carries the target color or the shape sequence on them is unintended, and the released wrapper stack does not filter these two keys out by default. A policy that flattens or concatenates the observation dictionary without filtering receives the answer for free on those 6 tasks, and any success rate so obtained measures nothing about retention. Drop both diagnostic keys explicitly before the observation reaches a policy (Appendix E describes how). Results on those 6 ids obtained without that filter should be discarded rather than reinterpreted: the shortcut is available at every step and leaves no trace in the score, so there is nothing in the number to correct. This paper’s own reference baseline is unaffected. It does not consume the observation dictionary at all: its prompt is assembled from a single per-dataset string, and its harness reads only the camera images and the discretized proprioceptive vector (Appendix H). Neither diagnostic channel ever reaches it. Action space. Actions are 7-dimensional end-effector commands (∆xyz,∆roll-pitch-yaw, gripper) under delta end-effector pose control, issued at10 Hz– the same 7 dimensions as the proprioceptive state. Not every channel is a delta. The gripper channel is an absolute mimic-joint target over the range[−0.01, 0.04] m, which is not symmetric about zero, so the zero vector does not mean “hold position”: after the normalization to[−1, 1]that both channels share, a stored zero commands the midpoint of that range and opens the fingers part-way. A policy that means “change nothing” has to predict the current gripper width, not a zero. Coordinate conventions. Positions and orientations are in the robot base frame, matching ManiSkill’s default convention for the underlying simulator. No additional coordinate transform is applied by the VLA wrapper stack beyond the joints-to-end-effector conversion itself. D Evaluation Protocol Section 5.2 describes the reference baseline’s evaluation in prose. The protocol it describes is not itself prose: the wrapper stack, the seeding, the episode count, and the success criterion are all fixed by the benchmark suite’s evaluation module, which is released with the code. Reproducing a number from this benchmark means running that module, not reimplementing a description of it. That module is a library rather than a runnable program: it supplies no command-line interface of its own. A separate example script, released alongside it, supplies the command line and a placeholder policy adapter to replace with one’s own. That script exposes options to choose which split and task to run, to override the start seed and episode count quoted below, to choose the simulator backend, to save videos, to choose an output directory, and to resume a partially completed run. 17 The first part of this appendix records what the released code fixes, so a reader can check the description against it rather than trust it. The second records where the reference baseline’s own evaluation departed from the released protocol, because the baseline was scored by the source training project’s evaluation code and not by this one. The third records the chance floor Section 5.2 reads that baseline against: what a policy that never looks at the observation scores on every task. D.1 What the released evaluation code fixes Wrapper stack. Every environment is constructed through one call to the released wrapper- construction function before any episode is stepped, with the same overlay setting (disabled) that produced the released episodes (Appendix E). A run therefore sees exactly the observation and action space of Appendix C, including the two diagnostic keys described there: there is no evaluation-only wrapper and no evaluation-only observation key. Debug overlays. Rendering the debug overlays and saving videos are both disabled by default in the evaluation module’s configuration, and the overlay wrappers are attached only when one of the two is switched on. Asking for videos therefore turns the overlays on as a side effect, so a run recorded for inspection is a visibly different configuration from the default one a reported number comes from. Note that this default is the opposite of the released wrapper-construction function’s own default (Appendix E). The evaluation module’s default is the one the release was collected under. Seeding and episode count. Both are fixed by the evaluation module: a start seed of4242424242 and 50 episodes per task, quoted here in full so that a reproduction can be checked against them without reading the source. A task’sk-th evaluation episode therefore carries the same seed on every run rather than a seed drawn afresh, and one episode count applies to every task in a run rather than a per-task choice. Because the simulator is deterministic given a seed (Appendix A), an independent group running the same start seed reproduces the same initial states, not merely the same distribution over them. The episode count is independent of the 250 episodes per task the dataset ships, which come from the collectors rather than from this evaluation module. The start-seed value is also reused elsewhere in the source as the wrapper stack’s missing-value sentinel (Appendix C). The two uses are unrelated, and a reader checking the source for that number will find both. Success criterion. An episode is scored by whether the task’s success condition held at any step of the episode, not only at the final step. This is the looser of the two readings, and it is the one behind every success rate in this paper, including the oracle success column of Table 33. A task whose success condition can be satisfied and then undone is credited for the step at which it was satisfied. D.2 How the reference baseline’s evaluation differed The reference baseline was not run through the released benchmark module. It was scored by the source training project’s own evaluation code, which builds each environment itself rather than through the released wrapper-construction function. That code transcribes the per-task wrapper chain out of the released wrapper-configuration table and cross-checks the transcription at build time against a separately hardcoded set of the task ids that should carry the curriculum wrapper, raising an error if the two disagree, so the chains cannot drift apart unnoticed. Six differences remain. We list them so that a reader who reproduces the baseline through the released code and gets a different figure knows which to look at first, rather than concluding the result does not reproduce. 1. An additional freeze wrapper on the-Long-variants. The baseline’s evaluation attaches the cue-phase freeze wrapper to the-Long-variants, holding the arm still through the cue phase – zero end-effector delta, gripper held open – which the released evaluation module does not attach to those variants by default. A reproduction through the released code should apply the same wrapper to match the baseline’s numbers reported here. 2. Render overlays on. The baseline’s evaluation attaches the task-specific overlays to every task, where the released default is off and the released episodes were collected with them off. Each overlay only changes what is drawn for its own rendering and leaves the rest of the environment step unchanged (Appendix E), so this changes the recorded video and neither the observation the policy receives nor the success it is scored on. 18 3.An extra outermost debug wrapper. The evaluation also mounts a reward-debugging wrapper that appears nowhere in the released wrapper-configuration table. It comes from the collector that recorded the PPO-sourced tasks, which mounted it outermost. It records the step’s internal state for its own rendering and leaves the rest of the step unchanged, so the previous item’s argument covers it. 4.A different seed lane and a smaller episode count. The baseline uses its own start seed, chosen to be disjoint from the seeds the training episodes were drawn from (Appendix H), and 20 episodes per task. Both differ from the start seed and episode count above, so the initial states behind the baseline’s numbers are a different set from the ones a default run visits, and a smaller one. 5.Episodes stop at the first success. The baseline’s rollout loop stops as soon as a step reports success, where the released code steps every episode to its horizon and takes the disjunction over steps. The success statistic is identical either way. The episode lengths are not, so any quantity derived from episode length, including wall-clock cost, will not match. 6.A split simulator backend. The baseline’s evaluation runs the motion-planned tasks on the CPU simulator backend and the PPO-sourced tasks on the GPU backend, splitting them by theSourcecolumn of Table 33, where the released evaluation default is the GPU backend for every task. The split is not arbitrary: it matches each task to the backend its own demonstrations were collected on, since the motion-planning collector builds its environments on the CPU backend and the collector behind the PPO-sourced tasks builds them on the GPU one. Items 2 and 3 are transparent to the score, since both wrappers leave the rest of the environment step unchanged. Item 5 is transparent for a different reason, and not one the wrappers can settle: the loop stops on the same condition the released path takes its disjunction over, so a run that stops early and a run that continues report the same success either way. Item 4 changes which episodes are run, so it can move a rate within the interval width of Section 6 without any of the numbers being wrong. Item 6 is the deviation this paper has not bounded. The two backends do not integrate identical actions into identical trajectories, so the split is not a matter of sampling and the interval width above says nothing about it. We ran no paired comparison that would give its size or its direction. A run on the released default is therefore a run under a different physics backend, and a disagreement there is not evidence against either number. Item 1 is the one that would make a reproduction disagree outright, and it disagrees in the released code’s favor until that code attaches the same wrapper. D.3 Chance floor: what a uniform-random policy scores Section 5.2 reads the reference baseline against a measured floor rather than an assumed one. This subsection is that measurement. Policy. Every action is drawn uniformly from the environment’s action space. The policy has no parameters and never reads the observation. Its action space is re-seeded at the start of each episode from the same seed as the environment reset, so a rerun visits the same trajectories. Protocol. All 90 canonical tasks, 50 episodes each, 4,500 episodes in total, from start seed 500,000. That lane is an independent draw rather than a repeat of any other run in this paper: it is disjoint from the evaluation module’s own start seed above, from the baseline’s 10,000 (Appendix H), and from the seeds the released demonstrations were collected under. Success uses the same any-step criterion as everywhere else in this paper. One deviation from the released stack. The released path builds every environment with overlays disabled, which requires image-plus-state observations. This run used state-only observations on the cpusimulator backend, because rendering 4,500 RGB rollouts was not possible in the window the measurement had: an unrelated job held every GPU on the machine at full utilization and near-full memory, and the renderer would not initialize. Two observation-processing layers of the released chain are therefore absent: the one that flattens the RGB-D observation into the released format, and the one that converts joint positions to end-effector pose. Neither can move a success statistic in this run, for a structural reason rather than an empirical one. Both only rewrite what the policy is shown, in different ways, and leave the action and the rest of the environment step unchanged (Appendix E). An observation-only layer can only change what a 19 policy sees, and this policy sees nothing. Neither layer touches the action, the physics, or the success condition. The one layer in the released chain that does modify actions, the cue-phase freeze wrapper, was applied on the 23 tasks the released wrapper-configuration table names and on no others. Result. Of the 4,500 episodes, 1 succeeded. Of the 90 tasks, 89 scored exactly zero. The remaining task,ShellGameShuffleTouch-Long-VLA-v0, scored 0.02, whose binomial standard error at n = 50is 0.020. Scored at the final step of the episode instead of at any step, the count over the same rollouts is 0: the one success was not held to the end of its episode. This bounds one thing and no more. Chance contributes nothing to a success rate on these tasks, so a low score is not an artifact of tasks that can be passed by accident. The floor does not show that any task requires memory, and it does not bound what a policy without retention could reach: a policy that manipulates competently and remembers nothing is not a random one, and nothing here measures one. A selection floor for the forgetful policy. That second policy can be bounded without running it, on the tasks whose success condition is a choice among a fixed candidate set. There the environment source fixes the number of candidatesk, so an arm that manipulates competently and has forgotten the cue still lands on the target with probability1/k. Of the 14 evaluated tasks, 7 are selections of this kind, withkequal to3,5, and6. The reference arm clears the floor on 0 of them: 5 of its rates fall within one standard error of1/kand the other 2 fall below. Pooled over the 140 episodes those tasks contribute, it solves 33 against the 35.3 a policy sitting exactly at each floor would average – a shortfall well inside the 5.1 that null’s own spread allows, so the arm is not below the floor either. It is indistinguishable from it. Which of the two readings this supports, a policy that has learned the manipulation and not the cue or a policy that is failing for some other reason, the comparison does not decide. Section 5.4 takes up the alternatives. Table 6: Selection floors for the 7 evaluated tasks whose success condition names a finite candidate set.kis the number of candidates the success condition selects among, never transcribed by hand. Success is the reference arm’s rate over 20 episodes. A rate below1/kdoes not mean the arm is worse than forgetful: at this episode count the two are not separable. Nor does a rate above it mean the arm is better – 0 of these rows clear their floor by more than the row’s own standard error. Taskk 1/kSuccess GatherAndRecall1-VLA-v030.3330.35 GatherAndRecall3-VLA-v030.3330.15 RememberColor5-Long-VLA-v050.2000.30 RememberColor5-VLA-v050.2000.25 RememberShape5-VLA-v050.2000.05 RememberShapeAndColor3x2-VLA-v060.1670.20 ShellGameShuffleColorLampTouch-Long-VLA-v030.3330.35 Table 6 gives each task’sk, its floor, and the arm’s rate. The tasks that get no floor are excluded for one of two reasons: some are continuous rather than selections, and the rest need several correct choices at once, where1/kdescribes no policy anyone could run. Three limits apply throughout. The floor bounds a hypothetical policy rather than any model that was run, and clearing it is not evidence of retention, since an observation shortcut would clear it too. On theGatherAndRecallfamily the bound is conditional on the manipulation succeeding first, which makes it weaker than the rest. And the rule declines to scoreBunchOfColors3-Long-VLA-v0andChainOfColors3-Long-VLA-v0, the evaluated tasks whose conditions enumerate the largest candidate sets, so every floor that survives is one of the highest available: a laxer rule would set a lower bar. Both score zero, under the1/k such a rule would have given them, so admitting them would not soften the comparison. Where the single success fell. OnShellGameShuffleTouch-Long-VLA-v0, one of the-Long- variants that the released evaluation module does not freeze through the cue phase by default. Its base variant carries the wrapper and scored zero. An arm that is not frozen through the cue phase has that phase in which to satisfy a success condition by accident, which is consistent with, though not proof of, why the one success landed there rather than on the base variant. One episode is far too few to attribute the success to the missing wrapper. Corroboration. An independent uniform-policy run made earlier on the benchmark’s Short split, from a different implementation and at the evaluation module’s own start seed rather than this lane, reports a success rate of 0.000526 over the 1,900 episodes that split holds. The matching figure from 20 this run is 0 successes in the same 1,900. That is the comparison: the 1-in-4,500 rate quoted above pools all three splits and is not the same quantity. Neither run resolves a rate this small. What the two agree on is that it is not distinguishable from zero. Software. ManiSkill 3.0.0b14, Gymnasium 0.29.1, PyTorch 2.2.1+cu121, Python 3.11.13. E Wrapper Cookbook Every task in this benchmark is registered as a standard Gymnasium environment. The VLA-facing observation and action space (Appendix C) is produced by applying one wrapper function immediately after the environment is constructed. The released benchmark’s own environment-construction helper performs the same steps internally, described here rather than listed as code. That helper is library code, invoked through a separate runnable example script (Appendix D). Constructing a task environment for this benchmark takes six explicit settings, none left to a simulator default: the number of parallel environments, the observation mode, the control mode, the render mode, the reward mode, and the simulator backend. The observation mode and control mode determine the shapes described in Appendix C; the reward mode selects the normalized dense reward Appendix B describes. The wrapper-construction function is then applied to that environment with overlays disabled, producing the released observation and action space. Resetting the environment with a given seed, sampling an action from the action space, and stepping the environment once yields an observation, a reward, a success flag, and the language instruction for that episode. Two observation keys must be dropped before the observation reaches a policy, as the next paragraph explains. The control mode in particular has no safe default here: the curriculum wrapper emits an action shaped for the delta end-effector pose control mode, and constructing the environment under any other control mode produces a runtime warning and a wrapper that no longer matches the action space. Two observation keys must be dropped before a policy sees them. Before filtering, the observation dictionary holds four keys, not two: the two uint8 camera images (top and wrist) at128× 128× 3 each, the 7-dimensional end-effector state, and two diagnostic channels carrying privileged state that the wrapper stack injects on every task. On 6 of the benchmark’s tasks, one of those diagnostic channels is set at reset and echoed unchanged on every step thereafter, so a policy fed the unfiltered dictionary can score well without retaining anything. The affected ids and the reason the guard is missing are in Appendix C. The instruction string is not part of the observation dictionary. It is returned separately, alongside the reward and success flag. The sampled action is a 7-dimensional delta end-effector pose. Overlays default on in the wrapper function, but off in the released evaluation. The wrapper- construction function defaults to rendering the debug overlays, but the released evaluation module’s configuration defaults that off, and the released episodes were collected with it off. A caller who reuses the wrapper function directly without disabling overlays therefore gets a configuration that is not the released one. Debug render overlays draw a step counter, live reward components, and task-specific state such as which cup currently hides the target. Each overlay only changes what is drawn for rendering, so it does not change what a policy is shown. Turning the overlays on therefore changes the rendered frame and nothing a policy sees. Video runs are the exception: the evaluation module switches to a frame-returning render mode when video saving is requested and turns the overlays on in the same branch, whatever the configured overlay flag says (Appendix D), so a run that writes videos is not built the same way a default run is. Both changes stay inside rendering: the observation mode decides the observation, and neither change reaches a policy. The wrapper table is the benchmark’s boundary. The environment registry is wider. Registering the benchmark’s environments registers several task ids that are not covered by the released wrapper- configuration table – a shell-game variant and several larger batteries-checker sizes among them – so constructing one of those environments succeeds on tasks the manifest does not list and this paper does not cover. Applying the wrapper-construction function to one of them raises an error naming the supported ids, rather than silently applying a default configuration that may not match the task, so the mismatch surfaces at construction rather than in the results. 21 F Per-Family Task Details Section 3.1 groups the 90 tasks into 10 memory types. Within a memory type, tasks further group into 26 environment families sharing the same underlying reward and success-condition code, at different variant sizes or horizons. Each subsection below covers one family: a motivating everyday analogy, its language instruction, task description, a representative rollout filmstrip, a table of every variant in the family, its verbatim success condition, and a summary of its dense reward structure, written from its compute_dense_reward source rather than quoted from it (Section 3.6). Every filmstrip in this appendix is sampled from the environment’s render channel, which draws debug text over the scene: a step counter, live reward components, and task-specific state such as which cup currently hides the target. None of it reaches the released observation, for two reasons the datasheet states in full (Appendix A): the overlay classes touchrenderand notstep, and collection ran with them switched off. F.1 Batteries Checker Easy Motivation.An electrician sorting a box of batteries pulled from old equipment tests each one in a socket, reads the result, and sets it aside – retesting a battery already confirmed working wastes time, and skipping one leaves the sort incomplete. “Find all working batteries by inserting each one into the socket, observing the lamp result, and then pressing the button to confirm.” Task. Sequential verification: insert batteries into a socket, observe the lamp result, remember which batteries work, and press the confirmation button. Figure 3: Rollout filmstrip, BatteriesCheckerEasy-3-VLA-v0 (a representative variant). Table 7: Batteries Checker Easy: variants, horizon, data source, median realized episode length, and information gap. Lengths and gaps are in environment steps. Every variant contributes 250 retained demonstrations, and their oracle success rate is 1.000. Each horizon’s split is given by Table 2. Env ID Horizon (steps)Source Median lengthGap BatteriesCheckerEasy-3-VLA-v0540MP363undetermined BatteriesCheckerEasy-6-VLA-v01080MP727undetermined Information gap: provenance.Every gap in the table above is read from the environment source. • BatteriesCheckerEasy-3-VLA-v0(gapundetermined), BatteriesCheckerEasy-6-VLA-v0(): The lamp cue (working/not) is revealed at insertion and stays visible for only 7 steps after a working hit. This variant returns the tested battery to its tray slot automatically rather than requiring the agent to carry it back, advancing to the confirmation stage on the first step that stage is reached. The button-press requirement therefore starts once the agent has inserted the battery and read the lamp, an interval timed by the agent’s own motion, not by a source constant. Success condition.Same criterion as the hard variant: all working batteries must be found through completed check-confirm cycles. Dense reward structure. The easy variant does not define its own dense reward. It reuses the hard variant’s shaping unchanged (see batteries_checker_hard below) – only the docstring, and the environment’s own auto-return-to-slot behavior, differ. 22 F.2 Batteries Checker Hard Motivation. The same sorting task at a larger box and a longer shift: the electrician has to hold a running tally of which batteries are already confirmed across many more items and a much longer interval before the sort is done. “Find all working batteries by inserting each one into the socket, observing the lamp result, returning it from the socket to its initial slot, and then pressing the button to confirm.” Task. Hard sequential verification: test each battery, remember the result, return it to its slot, and submit the working set. Figure 4: Rollout filmstrip, BatteriesCheckerHard-3-VLA-v0 (a representative variant). Table 8: Batteries Checker Hard: variants, horizon, data source, median realized episode length, and information gap. Lengths and gaps are in environment steps. Every variant contributes 250 retained demonstrations, and their oracle success rate is 1.000. Each horizon’s split is given by Table 2. Env ID Horizon (steps)Source Median lengthGap BatteriesCheckerHard-3-VLA-v01080MP699undetermined BatteriesCheckerHard-6-VLA-v02160MP1416undetermined Information gap: provenance.Every gap in the table above is read from the environment source. • BatteriesCheckerHard-3-VLA-v0(gapundetermined), BatteriesCheckerHard-6-VLA-v0(): The lamp cue (working/not) is revealed at insertion and stays visible for only 7 steps after a working hit, but the button-press requirement only starts once the battery is physically returned to its tray slot, a transition timed by the agent’s own motion, not a source constant. Success condition.Every working battery must be discovered through the full insert-return-confirm procedure. Partial progress does not count as success. Dense reward structure. A stage-gated sum keyed on the current action stage: during the insert stage, reach-toward-the-nearest-unchecked-battery plus carry-the-grasped-battery-toward-the-socket terms. During the return stage, a term that shrinks the distance between the active battery and its home tray slot. During the confirm stage, a button-reach plus button-press-depth term. On top of this, one-off bonuses fire on each new insert, return, and confirm event, a larger bonus fires the first time a battery is confirmed working, and a penalty applies to pressing the confirm button outside the confirm stage. Minus the smoothness penalty throughout, the whole stage-gated sum is masked to steps where the action is actually expected, then a fixed success bonus replaces it once every working battery has been found. F.3 Blink Count Button Press Motivation. Consumer electronics report internal fault codes by blinking a status LED a fixed number of times. A technician reading one has to count silently through the blink sequence and only act – pressing a diagnostic button the matching number of times – once the light goes solid. “Count how many times the blue lamp blinks, press the red button exactly that many times when the red lamp turns green, then press the black button to submit your answer.” 23 Task.Temporal counting: count blue-lamp blinks, wait for execution, press the red button exactly that many times, then submit. Figure 5: Rollout filmstrip, BlinkCountButtonPressEasy-VLA-v0 (a representative variant). Table 9: Blink Count Button Press: variants, horizon, data source, median realized episode length, and information gap. Lengths and gaps are in environment steps. Every variant contributes 250 retained demonstrations, and their oracle success rate is 1.000. Each horizon’s split is given by Table 2. Env ID Horizon (steps)Source Median lengthGap BlinkCountButtonPressEasy-VLA-v0150MP940 BlinkCountButtonPressMedium-VLA-v0200MP1210 BlinkCountButtonPressHard-VLA-v0300MP1530 BlinkCountButtonPressEasy-Long-VLA-v01200MP1990 BlinkCountButtonPressMedium-Long-VLA-v01200MP4900 BlinkCountButtonPressHard-Long-VLA-v01200MP7880 Information gap: provenance.Every gap in the table above is read from the environment source. • BlinkCountButtonPressEasy-VLA-v0(gap 0),BlinkCountButtonPressMedium-VLA-v0 (),BlinkCountButtonPressHard-VLA-v0(),BlinkCountButtonPressEasy-Long-VLA-v0 (),BlinkCountButtonPressMedium-Long-VLA-v0(), BlinkCountButtonPressHard-Long-VLA-v0():The manipulation requirement begins on the very step the cue phase ends, with no intervening empty phase: a 0-step gap. Success condition. • Success is produced only when the black submit button is pressed. •At submit time, the counted number of valid red-button presses must exactly match the target blink count. Dense reward structure. A phase-decomposed sum over four regimes (cue-watching, pre-lift press-and-count, post-press lift-and-release, and submit) rather than one continuous shaping term. During the cue it rewards staying near the button without pressing. During the count phase it rewards reach, press depth, and progress toward matching the target blink count, with a one-off bonus per new raw press. After each press it rewards lifting clear and releasing before the next press is allowed, with lift/hover penalties that discourage skipping the lift. Once the counted presses equal the target it rewards reaching and pressing the black confirm button. Early presses during the cue and holding the button down during the lift phase are both penalized. Minus the smoothness penalty throughout, a fixed penalty applies on an incorrect confirm. F.4 Bunch Of Colors Motivation. A shopper glances at a fruit display, notices which colors are represented, and only later, after being asked to fetch the colors you saw, has to act on that memory without the display still in view. “Observe which colored cubes appear during the cue, wait, then touch all of them in any order and press the center button.” Task. Set memory: observe a set of colored cubes, then touch all remembered colors in any order and press the center button. 24 Figure 6: Rollout filmstrip, BunchOfColors3-VLA-v0 (a representative variant). Table 10: Bunch Of Colors: variants, horizon, data source, median realized episode length, and information gap. Lengths and gaps are in environment steps. Every variant contributes 250 retained demonstrations, and their oracle success rate is 1.000. Each horizon’s split is given by Table 2. Env ID Horizon (steps)Source Median lengthGap BunchOfColors3-VLA-v0400MP1583 (2–4) BunchOfColors5-VLA-v0400MP2423 (2–4) BunchOfColors7-VLA-v0400MP3273 (2–4) BunchOfColors3-Long-VLA-v0700MP449225 (138–313) BunchOfColors5-Long-VLA-v0700MP524225 (138–313) BunchOfColors7-Long-VLA-v0700MP569225 (138–313) Information gap: provenance.Every gap in the table above is read from the environment source. • BunchOfColors3-VLA-v0(gap3(2–4)),BunchOfColors5-VLA-v0(), BunchOfColors7-VLA-v0(): The manipulation phase begins after an empty phase whose length is drawn uniformly between 1 and 5 steps at reset, so the gap is that draw itself. • BunchOfColors3-Long-VLA-v0(gap 225 (138–313)),BunchOfColors5-Long-VLA-v0 (),BunchOfColors7-Long-VLA-v0(): The manipulation phase begins after an empty phase whose length is drawn uniformly between 50 and 400 steps at reset, so the gap is that draw itself. Success condition. The robot must touch all target colors and avoid wrong selections. Dense reward structure. Reach-toward-goal shaping plus a count-based term that scales linearly with the fraction of the target color set touched so far (independent of order), a smaller bonus for being close to static, a flat penalty applies per wrong-color touch, and a displacement penalty discourages disturbing untouched cubes during the active (post-cue) phase. The whole reward is masked to the active phase and replaced by a fixed terminal value once every target color has been touched with no wrong touches. F.5 Chain Of Colors Motivation. The same glance at a display, but the shopper must reproduce not just which colors were there but the exact order they were arranged in – closer to memorizing a combination than a shopping list. “Observe which colored cubes appear during the cue, wait, then touch all of them in the same order as the cubes were shown and press the center button.” Task.Ordered sequence memory: observe colors in order, then touch the remembered colors in the same order before submission. Figure 7: Rollout filmstrip, ChainOfColors3-VLA-v0 (a representative variant). 25 Table 11: Chain Of Colors: variants, horizon, data source, median realized episode length, and information gap. Lengths and gaps are in environment steps. Every variant contributes 250 retained demonstrations, and their oracle success rate is 1.000. Each horizon’s split is given by Table 2. Env ID Horizon (steps)Source Median lengthGap ChainOfColors3-VLA-v0400MP1653 (2–4) ChainOfColors5-VLA-v0400MP2533 (2–4) ChainOfColors7-VLA-v0400MP3423 (2–4) ChainOfColors3-Long-VLA-v0800MP559225 (138–313) ChainOfColors5-Long-VLA-v01000MP740225 (138–313) ChainOfColors7-Long-VLA-v01200MP916225 (138–313) Information gap: provenance.Every gap in the table above is read from the environment source. • ChainOfColors3-VLA-v0(gap3(2–4)),ChainOfColors5-VLA-v0(), ChainOfColors7-VLA-v0(): The manipulation phase begins after an empty phase whose length is drawn uniformly between 1 and 5 steps at reset, so the gap is that draw itself. • ChainOfColors3-Long-VLA-v0(gap 225 (138–313)),ChainOfColors5-Long-VLA-v0 (),ChainOfColors7-Long-VLA-v0(): The manipulation phase begins after an empty phase whose length is drawn uniformly between 50 and 400 steps at reset, so the gap is that draw itself. Success condition. Every target color must be touched in the correct order, with no wrong or out-of-order selections. Dense reward structure. Structurally identical to bunch_of_colors – reach shaping, a per-touch count reward, a smaller bonus for being close to static, a wrong-touch penalty, and a cube- displacement penalty, all masked to the active phase with a fixed terminal bonus on success – except the count reward is computed from the length of the correctly *ordered* touch prefix (a cumulative-max of an elementwise match against the true sequence) rather than from set membership, so an out-of-order touch does not extend the credited count even if the color is correct. F.6 Find Imposter Color Motivation. A line inspector is shown a reference tray of parts, the tray is rearranged out of view, and the inspector has to find the one part now present that was not in the original set – comparing against a memory of the reference, not a live side-by-side. “Observe the cubes shown, wait, then touch the cube whose color was not present before.” Task. Out-of-set color detection: remember shown colors and touch the cube whose color was absent from the cue. Figure 8: Rollout filmstrip, FindImposterColor3-VLA-v0 (a representative variant). 26 Table 12: Find Imposter Color: variants, horizon, data source, median realized episode length, and information gap. Lengths and gaps are in environment steps. Every variant contributes 250 retained demonstrations, and their oracle success rate is 1.000. Each horizon’s split is given by Table 2. Env ID Horizon (steps)Source Median lengthGap FindImposterColor3-VLA-v025PPO153 (2–4) FindImposterColor5-VLA-v025PPO153 (2–4) FindImposterColor9-VLA-v025PPO163 (2–4) Information gap: provenance.Every gap in the table above is read from the environment source. • FindImposterColor3-VLA-v0(gap 3 (2–4)),FindImposterColor5-VLA-v0(), FindImposterColor9-VLA-v0(): The manipulation phase begins after an empty phase whose length is drawn uniformly between 1 and 5 steps at reset, so the gap is that draw itself. Success condition.TCP within GOAL_THRESH of the imposter cube in the manipulation phase. Dense reward structure.A single-phase reach-and-hold shaping term, saturating with the distance from the tool center point to the target cube, plus a smaller bonus for being close to static and for being static while the object is placed, minus the smoothness penalty. No separate distractor-avoidance term. The task is entirely encoded through which cube counts as the goal position. A fixed terminal bonus replaces the shaped reward on success. F.7 Find Imposter Shape Motivation.The same inspection task, but the odd part is identified by its geometry rather than its color – the kind of sorting-by-profile an inspector does on a line where parts are backlit or unpainted and color is not available as a cue. “Observe the shapes shown, wait, then touch the object whose shape was not present before.” Task. Out-of-set shape detection: remember shown shapes and touch the object whose shape was absent from the cue. Figure 9: Rollout filmstrip, FindImposterShape3-VLA-v0 (a representative variant). Table 13: Find Imposter Shape: variants, horizon, data source, median realized episode length, and information gap. Lengths and gaps are in environment steps. Every variant contributes 250 retained demonstrations, and their oracle success rate is 1.000. Each horizon’s split is given by Table 2. Env ID Horizon (steps)Source Median lengthGap FindImposterShape3-VLA-v025PPO143 (2–4) FindImposterShape5-VLA-v025PPO153 (2–4) FindImposterShape9-VLA-v025PPO153 (2–4) Information gap: provenance.Every gap in the table above is read from the environment source. 27 • FindImposterShape3-VLA-v0(gap 3 (2–4)),FindImposterShape5-VLA-v0(), FindImposterShape9-VLA-v0(): The manipulation phase begins after an empty phase whose length is drawn uniformly between 1 and 5 steps at reset, so the gap is that draw itself. Success condition.TCP within GOAL_THRESH of the imposter shape in the manipulation phase. Dense reward structure. Identical reward code to find_imposter_color: reach-to-goal shaping, a small static-robot bonus, and the smoothness penalty, with a fixed terminal bonus on success. F.8 Find Imposter Shape And Color Motivation.The inspection task again, now requiring both shape and color to match the memorized reference for a part to be accepted – a part that is the right shape in the wrong color, or vice versa, still counts as the imposter. “Observe the objects shown, wait, then touch the object whose shape and color combination was not present before.” Task. Out-of-set binding detection: remember shape-color combinations and touch the novel combination. Figure 10: Rollout filmstrip,FindImposterShapeAndColor3x2-VLA-v0(a representative vari- ant). Table 14: Find Imposter Shape And Color: variants, horizon, data source, median realized episode length, and information gap. Lengths and gaps are in environment steps. Every variant contributes 250 retained demonstrations, and their oracle success rate is 1.000. Each horizon’s split is given by Table 2. Env ID Horizon (steps)Source Median lengthGap FindImposterShapeAndColor3x2-VLA-v025PPO163 (2–4) FindImposterShapeAndColor3x3-VLA-v025PPO153 (2–4) FindImposterShapeAndColor5x3-VLA-v040PPO153 (2–4) Information gap: provenance.Every gap in the table above is read from the environment source. • FindImposterShapeAndColor3x2-VLA-v0(gap3(2–4)), FindImposterShapeAndColor3x3-VLA-v0(),FindImposterShapeAndColor5x3-VLA-v0 (): The manipulation phase begins after an empty phase whose length is drawn uniformly between 1 and 5 steps at reset, so the gap is that draw itself. Success condition.TCP within GOAL_THRESH of the imposter object in the manipulation phase. Dense reward structure. Identical reward code to find_imposter_color: reach-to-goal shaping, a small static-robot bonus, and the smoothness penalty, with a fixed terminal bonus on success. F.9 Gather And Recall Motivation. A warehouse picker is told to keep filling a cart from the shelves and, when a floor light flashes, immediately drop everything and deliver to the marked dock – the pending instruction has to survive an unrelated, longer primary task until its cue fires. 28 “Move all cubes onto the disc. A lamp will briefly flash while you work. After all cubes are placed, press the button matching the flash color.” Task.Concurrent manipulation plus cue recall: gather cubes while a lamp briefly flashes, then press the button matching the remembered color. Figure 11: Rollout filmstrip, GatherAndRecall1-VLA-v0 (a representative variant). Table 15: Gather And Recall: variants, horizon, data source, median realized episode length, and information gap. Lengths and gaps are in environment steps. Every variant contributes 250 retained demonstrations, and their oracle success rate is 1.000. Each horizon’s split is given by Table 2. Env ID Horizon (steps)Source Median lengthGap GatherAndRecall1-VLA-v0200MP115undetermined GatherAndRecall3-VLA-v0400MP318undetermined GatherAndRecall5-VLA-v0600MP514undetermined GatherAndRecall7-VLA-v0800MP721undetermined GatherAndRecall9-VLA-v01000MP900undetermined Information gap: provenance.Every gap in the table above is read from the environment source. • GatherAndRecall1-VLA-v0(gapundetermined),GatherAndRecall3-VLA-v0 (),GatherAndRecall5-VLA-v0(),GatherAndRecall7-VLA-v0(), GatherAndRecall9-VLA-v0(): The lamp flash triggers once a randomly drawn number of cubes are on the disc and lasts a randomly drawn duration between 8 and 14 steps, but the button-press requirement only starts once all cubes are on the disc – a transition timed by how fast the agent places the remaining cubes after the flash ends, not a source constant. Success condition. All cubes detected on the disc AND the correct color button is pressed. Dense reward structure. A concurrent-task sum: reach-to-nearest-unplaced-cube shaping, a placement-progress term proportional to the fraction of cubes already on the disc, and, only once every cube is on the disc, a reach term toward the button whose color matches the remembered lamp flash plus a press-depth term for that same button. Minus the smoothness penalty throughout. A fixed penalty applies when a wrong-color button is pressed after all cubes are placed, and a fixed success bonus replaces the shaped reward on success. F.10 Intercept Motivation. A part rolls down a conveyor toward the end of the line. A worker has to judge its path and reach the interception point before it falls off, acting on a trajectory rather than a fixed target location. “Intercept the rolling ball by moving to its path and deflecting it toward the target.” Task. Predictive interception: infer a rolling ball path and deflect it toward the target. Figure 12: Rollout filmstrip, InterceptSlow-VLA-v0 (a representative variant). 29 Table 16: Intercept: variants, horizon, data source, median realized episode length, and information gap. Lengths and gaps are in environment steps. Every variant contributes 250 retained demonstrations, and their oracle success rate is 1.000. Each horizon’s split is given by Table 2. Env ID Horizon (steps)Source Median lengthGap InterceptSlow-VLA-v060PPO43cue never hidden InterceptMedium-VLA-v060PPO41cue never hidden InterceptFast-VLA-v060PPO30cue never hidden Information gap: provenance.Every gap in the table above is read from the environment source. • InterceptSlow-VLA-v0(gap cue never hidden),InterceptMedium-VLA-v0(), InterceptFast-VLA-v0(): The ball is spawned moving and stays physically present and observable for the entire episode, with its position read directly at every step. There is no cue-hide phase or occlusion constant, so no cue-to-action retention gap exists to measure. Success condition. The ball center must end up inside the goal radius. Dense reward structure.Two-regime shaping switched by a sticky reached-target flag: before the tool center point first comes within a fixed distance of a hit-offset pose next to the moving ball, the reward is dominated by reach-to-hit-pose shaping. After that flag is set, the reward is dominated by a placement term that shrinks the ball’s xy distance to the goal region. A fixed terminal bonus replaces the shaped reward on success. Note this family has no explicit smoothness penalty term. F.11 Intercept Grab Motivation. The same rolling part now has to be caught and held rather than merely deflected – closer to fielding a dropped tool before it hits the floor than tapping it aside. “Intercept the rolling ball and grasp it to stop it.” Task. Predictive capture: infer a rolling ball path and grasp or stop it. Figure 13: Rollout filmstrip, InterceptGrabSlow-VLA-v0 (a representative variant). Table 17: Intercept Grab: variants, horizon, data source, median realized episode length, and information gap. Lengths and gaps are in environment steps. Every variant contributes 250 retained demonstrations, and their oracle success rate is 1.000. Each horizon’s split is given by Table 2. Env ID Horizon (steps)Source Median lengthGap InterceptGrabSlow-VLA-v060PPO24cue never hidden InterceptGrabMedium-VLA-v060PPO33cue never hidden InterceptGrabFast-VLA-v060PPO50cue never hidden Information gap: provenance.Every gap in the table above is read from the environment source. • InterceptGrabSlow-VLA-v0(gap cue never hidden),InterceptGrabMedium-VLA-v0 (),InterceptGrabFast-VLA-v0(): The docstring states explicitly ’the robot has no separate observation phase: it must react immediately’. The ball is continuously visible, so there is no cue-hide phase or occlusion constant and no retention gap to measure. 30 Success condition.The ball must be grasped and the robot must be static at the end of the episode step. Dense reward structure. A cascade of proximity- and grasp-state-gated bonuses rather than one shaping term: base reach-to-ball shaping, a flat bonus once the ball’s position is inside the gripper’s bounding geometry, an added proximity bump within a very close radius, an exponential-decay bonus that rewards closing the gripper toward the ball’s exact diameter once it is bracketed, another flat bonus once gripper width is near that target, and a large flat bonus once the ball is grasped. A still-larger fixed bonus replaces the shaped reward on success. Two computed static-robot bonus quantities are never added into the returned reward – unused in the source, not part of the actual shaping. F.12 Remember Color Motivation. A sorter is shown one reference cube’s color, the cube is taken away, and several identical-looking cubes of different colors are set out – the sorter has to pick the one matching a color no longer visible. “Observe the cube’s color, wait, then touch the cube of the same color.” Task.Delayed color recall: observe one target color, wait, and touch the cube with the same color. Figure 14: Rollout filmstrip, RememberColor3-VLA-v0 (a representative variant). Table 18: Remember Color: variants, horizon, data source, median realized episode length, and information gap. Lengths and gaps are in environment steps. Every variant contributes 250 retained demonstrations, and their oracle success rate is 1.000. Each horizon’s split is given by Table 2. Env ID Horizon (steps)Source Median lengthGap RememberColor3-VLA-v025PPO153 (2–4) RememberColor5-VLA-v025PPO153 (2–4) RememberColor9-VLA-v025PPO153 (2–4) RememberColor3-Long-VLA-v0600MP337250 (150–350) RememberColor5-Long-VLA-v0600MP390250 (150–350) RememberColor9-Long-VLA-v0600MP378250 (150–350) Information gap: provenance.Every gap in the table above is read from the environment source. • RememberColor3-VLA-v0(gap3(2–4)),RememberColor5-VLA-v0(), RememberColor9-VLA-v0(): The manipulation phase begins after an empty phase whose length is drawn uniformly between 1 and 5 steps at reset, so the gap is that draw itself. • RememberColor3-Long-VLA-v0(gap 250 (150–350)),RememberColor5-Long-VLA-v0 (),RememberColor9-Long-VLA-v0(): The manipulation phase begins after an empty phase whose length is drawn uniformly between 50 and 450 steps at reset, so the gap is that draw itself. Success condition. The robot must reach the cube whose color matches the cue and satisfy the environment reach threshold. Dense reward structure. Identical reward code to find_imposter_color: reach-to-goal shaping, a small static-robot bonus, and the smoothness penalty, with a fixed terminal bonus on success. 31 F.13 Remember Shape Motivation. The same sorting task by geometry instead of color, the way a machinist might be shown a reference bolt head and then asked to find its match among a mixed bin by feel and shape alone. “Observe the object’s shape, wait, then touch the object of the same shape.” Task. Delayed shape recall: observe one target shape, wait, and touch the object with the same shape. Figure 15: Rollout filmstrip, RememberShape3-VLA-v0 (a representative variant). Table 19: Remember Shape: variants, horizon, data source, median realized episode length, and information gap. Lengths and gaps are in environment steps. Every variant contributes 250 retained demonstrations, and their oracle success rate is 1.000. Each horizon’s split is given by Table 2. Env ID Horizon (steps)Source Median lengthGap RememberShape3-VLA-v025PPO153 (2–4) RememberShape5-VLA-v025PPO153 (2–4) RememberShape9-VLA-v025PPO153 (2–4) RememberShape3-Long-VLA-v0600MP337250 (150–350) RememberShape5-Long-VLA-v0600MP349250 (150–350) RememberShape9-Long-VLA-v0600MP328250 (150–350) Information gap: provenance.Every gap in the table above is read from the environment source. • RememberShape3-VLA-v0(gap3(2–4)),RememberShape5-VLA-v0(), RememberShape9-VLA-v0(): The manipulation phase begins after an empty phase whose length is drawn uniformly between 1 and 5 steps at reset, so the gap is that draw itself. • RememberShape3-Long-VLA-v0(gap 250 (150–350)),RememberShape5-Long-VLA-v0 (),RememberShape9-Long-VLA-v0(): The manipulation phase begins after an empty phase whose length is drawn uniformly between 50 and 450 steps at reset, so the gap is that draw itself. Success condition. The robot must reach the object whose geometry matches the cue. Dense reward structure. Identical reward code to find_imposter_color: reach-to-goal shaping, a small static-robot bonus, and the smoothness penalty, with a fixed terminal bonus on success. F.14 Remember Shape And Color Motivation. The sorting task once more, now requiring both shape and color to match the item shown moments earlier – a distractor sharing only one attribute is not close enough. “Observe the object’s shape and color, wait, then touch the object of the same shape and color.” Task. Delayed binding recall: remember both shape and color and select the matching object. Figure 16: Rollout filmstrip, RememberShapeAndColor3x2-VLA-v0 (a representative variant). 32 Table 20: Remember Shape And Color: variants, horizon, data source, median realized episode length, and information gap. Lengths and gaps are in environment steps. Every variant contributes 250 retained demonstrations, and their oracle success rate is 1.000. Each horizon’s split is given by Table 2. Env ID Horizon (steps)Source Median lengthGap RememberShapeAndColor3x2-VLA-v025PPO153 (2–4) RememberShapeAndColor3x3-VLA-v025PPO143 (2–4) RememberShapeAndColor5x3-VLA-v025PPO153 (2–4) RememberShapeAndColor3x2-Long-VLA-v0600MP312250 (150–350) RememberShapeAndColor3x3-Long-VLA-v0600MP321250 (150–350) RememberShapeAndColor5x3-Long-VLA-v0600MP322250 (150–350) Information gap: provenance.Every gap in the table above is read from the environment source. • RememberShapeAndColor3x2-VLA-v0(gap3(2–4)), RememberShapeAndColor3x3-VLA-v0(),RememberShapeAndColor5x3-VLA-v0 (): The manipulation phase begins after an empty phase whose length is drawn uniformly between 1 and 5 steps at reset, so the gap is that draw itself. • RememberShapeAndColor3x2-Long-VLA-v0(gap250(150– 350)),RememberShapeAndColor3x3-Long-VLA-v0(), RememberShapeAndColor5x3-Long-VLA-v0():The manipulation phase begins after an empty phase whose length is drawn uniformly between 50 and 450 steps at reset, so the gap is that draw itself. Success condition. The robot must reach the object whose shape and color both match the cue. Dense reward structure. Identical reward code to find_imposter_color: reach-to-goal shaping, a small static-robot bonus, and the smoothness penalty, with a fixed terminal bonus on success. F.15 Rotate Lenient Motivation.A technician is told to turn a dial or connector to roughly a marked angle from memory, the way a valve gets set to about a quarter turn without a protractor in hand. “Rotate the peg by angle_deg degrees to match the target angle.” The braces are a placeholder: the environment substitutes that episode’s own target angle, as the provenance below records. Task. Spatial transformation: rotate a peg by a requested angle with a lenient center-position criterion. Figure 17: Rollout filmstrip, RotateLenientPos-VLA-v0 (a representative variant). Table 21: Rotate Lenient: variants, horizon, data source, median realized episode length, and information gap. Lengths and gaps are in environment steps. Every variant contributes 250 retained demonstrations, and their oracle success rate is 1.000. Each horizon’s split is given by Table 2. Env ID Horizon (steps)Source Median lengthGap RotateLenientPos-VLA-v060PPO29cue never hidden RotateLenientPosNeg-VLA-v060PPO19cue never hidden 33 Information gap: provenance.Every gap in the table above is read from the environment source. • RotateLenientPos-VLA-v0(gap cue never hidden),RotateLenientPosNeg-VLA-v0 (): The target angle is set once at reset and never hidden. The language instruction is recomputed from the target angle at every step, so the target is available continuously via the prompt rather than through a fading visual cue, and there is no cue-hide phase to measure a retention gap over. Success condition. The peg angle must be withinangle_thresholdof the target and the robot must be static. Dense reward structure. Exponential reach-to-peg shaping plus the smoothness penalty apply throughout. Once the tool center point is within a reach threshold, an exponential-in-angle-error term is added, and flat bonuses are added once the angle is within tolerance and again if the robot is also static at that point. A fixed terminal bonus replaces the shaped reward on success. F.16 Rotate Strict Motivation. The same rotation task, but now with a narrow tolerance – like aligning a keyed connector that only seats correctly within a few degrees of the specified angle, where an approximate quarter turn is not good enough. “Rotate the peg by angle_deg degrees to match the target angle while keeping the center of the peg in place.” The braces are a placeholder: the environment substitutes that episode’s own target angle, as the provenance below records. Task. Spatial transformation: rotate a peg while keeping its center close to the original position. Figure 18: Rollout filmstrip, RotateStrictPos-VLA-v0 (a representative variant). Table 22: Rotate Strict: variants, horizon, data source, median realized episode length, and information gap. Lengths and gaps are in environment steps. Every variant contributes 250 retained demonstrations, and their oracle success rate is 1.000. Each horizon’s split is given by Table 2. Env ID Horizon (steps)Source Median lengthGap RotateStrictPos-VLA-v090PPO31cue never hidden RotateStrictPosNeg-VLA-v090PPO22cue never hidden Information gap: provenance.Every gap in the table above is read from the environment source. • RotateStrictPos-VLA-v0(gap cue never hidden),RotateStrictPosNeg-VLA-v0(): The target angle is set once at reset and never hidden. The language instruction is recomputed from the target angle at every step, so the target is available continuously via the prompt rather than through a fading visual cue, and there is no cue-hide phase to measure a retention gap over. Success condition. The peg angle must be withinangle_threshold, the peg must stay close to its initial XY position, and the robot must be static. 34 Dense reward structure. The same reach-then-angle shaping as rotate_lenient, but gated more strictly: once reached, x/y position-holding terms are added (rewarding keeping the peg’s xy position near its start), and the large angle-matching bonus and the two flat tolerance/static bonuses only fire once the peg is both reached *and* within xy position tolerance – reflecting that this variant additionally requires the peg to stay in place while rotating. A larger fixed terminal bonus replaces the shaped reward on success. F.17 Seq Of Colors Motivation.A worker observes a sequence of color-coded items and must later reproduce the same order while avoiding distractors. This requires retaining both the members of the set and their order. “Observe which colored cubes appear during the cue, wait, then touch all of them in any order and press the center button.” Task. Sequential cue, unordered execution: colors are shown sequentially, but the remembered targets can be touched in any order. Figure 19: Rollout filmstrip, SeqOfColors3-VLA-v0 (a representative variant). Table 23: Seq Of Colors: variants, horizon, data source, median realized episode length, and information gap. Lengths and gaps are in environment steps. Every variant contributes 250 retained demonstrations, and their oracle success rate is 1.000. Each horizon’s split is given by Table 2. Env ID Horizon (steps)Source Median lengthGap SeqOfColors3-VLA-v0400MP1653 (2–4) SeqOfColors5-VLA-v0400MP2533 (2–4) SeqOfColors7-VLA-v0400MP3423 (2–4) SeqOfColors3-Long-VLA-v0800MP559225 (138–313) SeqOfColors5-Long-VLA-v01000MP738225 (138–313) SeqOfColors7-Long-VLA-v01200MP916225 (138–313) Information gap: provenance.Every gap in the table above is read from the environment source. • SeqOfColors3-VLA-v0(gap3(2–4)),SeqOfColors5-VLA-v0(), SeqOfColors7-VLA-v0(): The manipulation phase begins after an empty phase whose length is drawn uniformly between 1 and 5 steps at reset, so the gap is that draw itself. • SeqOfColors3-Long-VLA-v0(gap 225 (138–313)),SeqOfColors5-Long-VLA-v0(), SeqOfColors7-Long-VLA-v0(): The manipulation phase begins after an empty phase whose length is drawn uniformly between 50 and 400 steps at reset, so the gap is that draw itself. Success condition.The robot must touch every color that appeared in the cue sequence and avoid touching colors that were not part of that sequence. The docstring describes set membership only. The implemented success condition additionally requires the touch order to match the cue order. The released instruction has the same mismatch, which Section 6 records. Dense reward structure.The shaping is shared with bunch_of_colors: reach reward, a count-based term for touching cued colors, a static bonus, and penalties for wrong touches and cube displacement, all masked to the active phase. Terminal success receives a fixed bonus. This progress term is order-insensitive, whereas the executable success condition requires the cue order. The released instruction says that any order is allowed. 35 F.18 Shell Game Color Lamp Touch Motivation. A variant of the classic shell game where the winning cup is revealed by a lamp’s color rather than which cup moved – the player has to remember a color-to-cup mapping shown once, then act on it after the mapping is no longer displayed. “Observe which color is under each cup, then touch the cup matching the lamp color.” Task. Hidden color-location binding: remember which color is under each cup and touch the cup matching the lamp color. Figure 20: Rollout filmstrip, ShellGameColorLampTouch-VLA-v0 (a representative variant). Table 24: Shell Game Color Lamp Touch: variants, horizon, data source, median realized episode length, and information gap. Lengths and gaps are in environment steps. Every variant contributes 250 retained demonstrations, and their oracle success rate is 1.000. Each horizon’s split is given by Table 2. Env ID Horizon (steps)Source Median lengthGap ShellGameColorLampTouch-VLA-v030PPO150 Information gap: provenance.Every gap in the table above is read from the environment source. • ShellGameColorLampTouch-VLA-v0(gap 0): The manipulation requirement begins on the very step the cue phase ends, with no intervening empty phase: a 0-step gap. Success condition.The robot must touch the cup covering the ball whose color matches the lamp, and the robot must be static. Dense reward structure. The same reach-plus-static shaping as shell_game_touch, but without a mug-shift penalty term – the source cups are not perturbed after the cue in this variant. Masked to the manipulation phase, with a fixed terminal bonus on success. F.19 Shell Game Push Motivation.The classic shell game, but the found cup must be pushed aside rather than touched – the same memory of which cup hides the target, expressed through a different action. “Observe which cup hides the ball, wait, then push that cup forward.” Task. Hidden object selection: remember which cup hides the ball and push that cup forward. Figure 21: Rollout filmstrip, ShellGamePush-VLA-v0 (a representative variant). 36 Table 25: Shell Game Push: variants, horizon, data source, median realized episode length, and information gap. Lengths and gaps are in environment steps. Every variant contributes 250 retained demonstrations, and their oracle success rate is 1.000. Each horizon’s split is given by Table 2. Env ID Horizon (steps)Source Median lengthGap ShellGamePush-VLA-v030PPO150 Information gap: provenance.Every gap in the table above is read from the environment source. • ShellGamePush-VLA-v0(gap 0): The touch/push requirement begins on the very step the cue (ball position) is hidden, with no intervening empty phase: a 0-step gap. Success condition. The ball center must finish inside the goal region. Dense reward structure. Reach-to-ball shaping throughout, plus a place-toward-goal term that only activates once the tool center point has reached the ball, plus small static-robot bonuses once the ball is placed. Fixed terminal bonus on success. No explicit penalty term for disturbing the other cups. F.20 Shell Game Shuffle Color Lamp Touch Motivation. The color-lamp shell game with the added twist that the cups are physically shuffled after the mapping is shown, so the player has to track a moving target rather than recall a single static fact. “Observe which color is under each cup, track the cups as they shuffle, then touch the cup matching the lamp color.” Task.Hidden color-location tracking: track cup-color associations through shuffling and touch the lamp-matching cup. Figure 22: Rollout filmstrip,ShellGameShuffleColorLampTouch-VLA-v0(a representative variant). Table 26: Shell Game Shuffle Color Lamp Touch: variants, horizon, data source, median realized episode length, and information gap. Lengths and gaps are in environment steps. Every variant contributes 250 retained demonstrations, and their oracle success rate is 1.000. Each horizon’s split is given by Table 2. Env ID Horizon (steps)Source Median lengthGap ShellGameShuffleColorLampTouch-VLA-v060PPO4428 (24–31) ShellGameShuffleColorLampTouch-Long-VLA-v0600MP331250 (175–325) Information gap: provenance.Every gap in the table above is read from the environment source. • ShellGameShuffleColorLampTouch-VLA-v0(gap 28 (24–31)): The manipulation phase begins after a shuffle phase whose length is drawn uniformly between 20 and 35 steps at reset. • ShellGameShuffleColorLampTouch-Long-VLA-v0(gap 250 (175–325)): The manipu- lation phase begins after a shuffle phase whose length is drawn uniformly between 100 and 400 steps at reset. 37 Success condition.The robot must touch the cup that hides the ball whose color matches the final lamp cue. Dense reward structure. Identical reward code to shell_game_color_lamp_touch: reach-plus- static shaping with no mug-shift penalty, masked to the manipulation phase, fixed terminal bonus on success. F.21 Shell Game Shuffle Touch Motivation.The classic shell game played the way it is actually hustled on a street corner: the cups are shuffled after the ball is hidden, so the player has to visually track the target through the shuffle rather than remember its starting position. “Observe which cup hides the ball, track the cups as they shuffle, then touch the correct cup.” Task. Hidden object tracking: track the ball-hiding cup through shuffles and touch it. Figure 23: Rollout filmstrip, ShellGameShuffleTouch-VLA-v0 (a representative variant). Table 27: Shell Game Shuffle Touch: variants, horizon, data source, median realized episode length, and information gap. Lengths and gaps are in environment steps. Every variant contributes 250 retained demonstrations, and their oracle success rate is 1.000. Each horizon’s split is given by Table 2. Env ID Horizon (steps)Source Median lengthGap ShellGameShuffleTouch-VLA-v060PPO4228 (24–31) ShellGameShuffleTouch-Long-VLA-v0600MP337250 (175–325) Information gap: provenance.Every gap in the table above is read from the environment source. • ShellGameShuffleTouch-VLA-v0(gap 28 (24–31)): The manipulation phase begins after a shuffle phase whose length is drawn uniformly between 20 and 35 steps at reset. • ShellGameShuffleTouch-Long-VLA-v0 (gap 250 (175–325)): The manipulation phase begins after a shuffle phase whose length is drawn uniformly between 100 and 400 steps at reset. Success condition. The robot must touch the final cup position that contains the hidden ball. Dense reward structure. Identical reward code to shell_game_touch, including the mug-shift penalty on the largest displacement across all three cups (touched cup included, not excluded). The only functional difference from shell_game_touch is the environment’s own shuffle phase before manipulation begins, not the reward shaping itself. F.22 Shell Game Touch Motivation. The simplest form of the shell game: the target is hidden once, the cups do not move afterward, and the player only has to remember where it went rather than track it moving. “Observe which cup hides the ball, wait, then touch that cup.” 38 Task. Short hidden object memory: observe which cup hides the ball, wait, and touch that cup. Figure 24: Rollout filmstrip, ShellGameTouch-VLA-v0 (a representative variant). Table 28: Shell Game Touch: variants, horizon, data source, median realized episode length, and information gap. Lengths and gaps are in environment steps. Every variant contributes 250 retained demonstrations, and their oracle success rate is 1.000. Each horizon’s split is given by Table 2. Env ID Horizon (steps)Source Median lengthGap ShellGameTouch-VLA-v030PPO220 Information gap: provenance.Every gap in the table above is read from the environment source. • ShellGameTouch-VLA-v0(gap 0): The touch/push requirement begins on the very step the cue (ball position) is hidden, with no intervening empty phase: a 0-step gap. Success condition. The robot must touch the cup covering the memorized ball location. Dense reward structure.Reach-to-goal shaping plus small bonuses for being static once the touch is registered, minus a cup-displacement penalty (saturating with the largest displacement among all three cups – left, center, and right – with no exclusion for whichever cup is being touched) that discourages disturbing any cup during manipulation. Masked to the manipulation phase, with a fixed terminal bonus on success. F.23 Take It Back Motivation. A librarian pulls a book from the shelf to check something, sets it down elsewhere, and afterward has to return it to its exact original slot – not just any open space on the shelf, but the specific spot it came from. “Push the cube onto the red target, and when the target changes color, return the cube to its original position.” Task. Spatial restoration: push a cube to a target, then return it to its original position after the target changes. Figure 25: Rollout filmstrip, TakeItBack-VLA-v0 (a representative variant). Table 29: Take It Back: variants, horizon, data source, median realized episode length, and information gap. Lengths and gaps are in environment steps. Every variant contributes 250 retained demonstrations, and their oracle success rate is 1.000. Each horizon’s split is given by Table 2. Env ID Horizon (steps)Source Median lengthGap TakeItBack-VLA-v060PPO27undetermined 39 Information gap: provenance.Every gap in the table above is read from the environment source. • TakeItBack-VLA-v0(gap undetermined): The initial region is treated as a hidden object and never rendered, and the return stage begins only once the first goal is achieved – an event timed by how fast the agent completes the first push, not a source constant, so the gap cannot be read off as a constant or a sampling range. Success condition.The cube must first reach the goal region and then end up back inside the initial region within the same episode. Dense reward structure. A two-stage push-then-pull sum switched by whether the cube has already reached the goal region: before that, the reward is dominated by reach-to-push-pose and place-in-goal shaping, with an added bonus once positioned to push. After that, it is dominated by reach-to-pull-pose and keep-contact shaping, with an added bonus once positioned to pull and place the cube back at its initial region. Minus the smoothness penalty throughout, fixed terminal bonus on success. F.24 Timed Transfer Motivation. A relay handoff on an assembly line has to happen inside a specific timing window signaled earlier – handing off too early or too late both miss the window, the way a baton pass only counts inside a marked zone. “When the white lamp turns green, start counting steps from that exact moment. Move the blue cube from the green disc to the red disc exactly on step 100 of that count.” Task. Delayed timing: after a lamp cue, move the cube at a precise future timestep. Figure 26: Rollout filmstrip, TimedTransferEasy-VLA-v0 (a representative variant). Table 30: Timed Transfer: variants, horizon, data source, median realized episode length, and information gap. Lengths and gaps are in environment steps. Every variant contributes 250 retained demonstrations, and their oracle success rate is 1.000. Each horizon’s split is given by Table 2. Env ID Horizon (steps)Source Median lengthGap TimedTransferEasy-VLA-v0200MP106100 TimedTransferMedium-VLA-v0250MP152150 TimedTransferHard-VLA-v0300MP203200 TimedTransferEasy-Long-VLA-v0600MP298300 TimedTransferMedium-Long-VLA-v0900MP488500 TimedTransferHard-Long-VLA-v01200MP9621000 Information gap: provenance.Every gap in the table above is read from the environment source. • TimedTransferEasy-VLA-v0(gap 100): The required delay is a fixed 100 steps (not resampled per episode). The cube must land on the red disc within a fixed tolerance of that many steps after the signal, so the required retention interval is exactly 100 steps. • TimedTransferMedium-VLA-v0 (gap 150): The required delay is a fixed 150 steps (not resampled per episode). The cube must land on the red disc within a fixed tolerance of that many steps after the signal, so the required retention interval is exactly 150 steps. • TimedTransferHard-VLA-v0(gap 200): The required delay is a fixed 200 steps (not resampled per episode). The cube must land on the red disc within a fixed tolerance of that many steps after the signal, so the required retention interval is exactly 200 steps. 40 • TimedTransferEasy-Long-VLA-v0(gap 300): The required delay is a fixed 300 steps (not resampled per episode). The cube must land on the red disc within a fixed tolerance of that many steps after the signal, so the required retention interval is exactly 300 steps. • TimedTransferMedium-Long-VLA-v0 (gap 500): The required delay is a fixed 500 steps (not resampled per episode). The cube must land on the red disc within a fixed tolerance of that many steps after the signal, so the required retention interval is exactly 500 steps. • TimedTransferHard-Long-VLA-v0(gap 1000): The required delay is a fixed 1000 steps (not resampled per episode). The cube must land on the red disc within a fixed tolerance of that many steps after the signal, so the required retention interval is exactly 1000 steps. Success condition (from the source docstring’s episode-flow description). • The lamp is off. The blue cube sits on the green disc. • After a brief random delay the lamp turns green (the signal). • The agent must internally count DELAY_STEPS from the signal. • The agent picks the blue cube and places it on the red disc. •The cube must be on the red disc within +/-TOLERANCE_FRAC of the target step. Placing too early or too late is a failure. Dense reward structure. A timing state machine gates the reward across waiting and transfer phases. Reward is zero before the cue lamp turns on. Afterward, reach and grasp bonuses accrue. Placement and static-hold bonuses are available only in the target step window, while early placement is penalized. A smoothness penalty applies throughout, with fixed failure and success terms. This family tests elapsed-time tracking, rather than retention of a hidden visual cue. F.25 Trace Shape Motivation. A physical therapist demonstrates a movement pattern once and asks the patient to reproduce it without a live reference to copy – the trajectory itself, not just its start and end points, has to be remembered. “Watch the red cube trace a shape on the table. When the lamp turns green, pick up the green cube and trace exactly the same shape.” Task. Demonstration imitation: watch a cube trace a shape, then reproduce the path. Figure 27: Rollout filmstrip, TraceShapeEasy-VLA-v0 (a representative variant). Table 31: Trace Shape: variants, horizon, data source, median realized episode length, and information gap. Lengths and gaps are in environment steps. Every variant contributes 250 retained demonstrations, and their oracle success rate is 1.000. Each horizon’s split is given by Table 2. Env ID Horizon (steps)Source Median lengthGap TraceShapeEasy-VLA-v0250MP2090 TraceShapeMedium-VLA-v0300MP2100 TraceShapeHard-VLA-v0350MP2070 Information gap: provenance.Every gap in the table above is read from the environment source. • TraceShapeEasy-VLA-v0(gap0),TraceShapeMedium-VLA-v0(), TraceShapeHard-VLA-v0(): The manipulation requirement begins as soon as the cue phase elapses, with no separate empty phase. The red-cube demo is hidden the same step the manipulation requirement starts: a 0-step gap. 41 Success condition. • The green cube must visit every checkpoint along the demonstrated path. • After that, the contour must be explicitly closed by returning near the first checkpoint (start point) within CHECKPOINT_THRESH. Dense reward structure.A checkpoint-tracing sum: a small reach term during the cue (watching the demonstration), then during the action phase, reach and grasp-indicator terms, a term proportional to the fraction of checkpoints visited so far, a nearest-unvisited-checkpoint distance term gated on already grasping the cube, and – once every checkpoint has been visited – a closure term that rewards returning near the starting checkpoint. Minus the smoothness penalty, fixed terminal bonus on success. F.26 Trace Shape Seq Motivation. The same demonstrated-motion task, now chaining several distinct shapes shown in sequence, so the reproduction has to preserve both each shape and the order they were shown in. “Watch the red cube trace a sequence of shapes. When the lamp turns green, pick up the green cube and trace the same sequence in order. After finishing all shapes, press the button to submit your answer.” Task. Long-horizon demonstration imitation: watch and reproduce a sequence of traced shapes. Figure 28: Rollout filmstrip, TraceShapeSeqEasy-VLA-v0 (a representative variant). Table 32: Trace Shape Seq: variants, horizon, data source, median realized episode length, and information gap. Lengths and gaps are in environment steps. Every variant contributes 250 retained demonstrations, and their oracle success rate is 1.000. Each horizon’s split is given by Table 2. Env ID Horizon (steps)Source Median lengthGap TraceShapeSeqEasy-VLA-v01500MP7020 TraceShapeSeqMedium-VLA-v01500MP7200 TraceShapeSeqHard-VLA-v01500MP7060 Information gap: provenance.Every gap in the table above is read from the environment source. • TraceShapeSeqEasy-VLA-v0(gap0),TraceShapeSeqMedium-VLA-v0(), TraceShapeSeqHard-VLA-v0(): The manipulation requirement begins as soon as the cue phase elapses, with no separate empty phase. The red-cube demo is hidden the same step the manipulation requirement starts: a 0-step gap. Success condition. • Every sequence element must be completed in order. • A sequence element is complete only when all its checkpoints are visited and the contour is closed (return near checkpoint[0]). • After all elements are complete, the robot must press the button. Dense reward structure.Extends trace_shape’s checkpoint-tracing shaping (reach, grasp indicator, per-shape visit fraction, nearest-checkpoint term, closure term) with a sequence-progress term that credits completing the shapes of the sequence in order, and, once every shape in the sequence is closed, a reach-then-press term toward a submit button. A fixed penalty applies on an incorrect or otherwise failed submit, fixed terminal bonus on success. 42 G Full Per-Task Statistics Table 33 below lists all 90 canonical tasks. 1 Figure 29 ranks those tasks by total released frames. 1 The task registry also lists 5*-Medium-VLA-v0environment ids beyond these 90. They belong to a different project, not to MIKASA-Robo-VLA, and are excluded from every count, table, and figure in this paper. 43 050000100000150000200000250000300000350000 Total frames RememberShapeAndColor3x3-VLA-v0 FindImposterShape3-VLA-v0 RememberShape3-VLA-v0 FindImposterColor5-VLA-v0 ShellGamePush-VLA-v0 RememberShapeAndColor3x2-VLA-v0 RememberColor9-VLA-v0 RememberShape5-VLA-v0 FindImposterShapeAndColor3x3-VLA-v0 FindImposterShape9-VLA-v0 FindImposterColor3-VLA-v0 RememberColor5-VLA-v0 RememberShapeAndColor5x3-VLA-v0 FindImposterShape5-VLA-v0 ShellGameColorLampTouch-VLA-v0 RememberColor3-VLA-v0 RememberShape9-VLA-v0 FindImposterColor9-VLA-v0 FindImposterShapeAndColor3x2-VLA-v0 FindImposterShapeAndColor5x3-VLA-v0 RotateLenientPosNeg-VLA-v0 ShellGameTouch-VLA-v0 RotateStrictPosNeg-VLA-v0 InterceptGrabSlow-VLA-v0 TakeItBack-VLA-v0 RotateLenientPos-VLA-v0 InterceptFast-VLA-v0 RotateStrictPos-VLA-v0 InterceptGrabMedium-VLA-v0 InterceptMedium-VLA-v0 ShellGameShuffleTouch-VLA-v0 ShellGameShuffleColorLampTouch-VLA-v0 InterceptSlow-VLA-v0 InterceptGrabFast-VLA-v0 BlinkCountButtonPressEasy-VLA-v0 TimedTransferEasy-VLA-v0 GatherAndRecall1-VLA-v0 BlinkCountButtonPressMedium-VLA-v0 BlinkCountButtonPressHard-VLA-v0 TimedTransferMedium-VLA-v0 BunchOfColors3-VLA-v0 ChainOfColors3-VLA-v0 SeqOfColors3-VLA-v0 TimedTransferHard-VLA-v0 BlinkCountButtonPressEasy-Long-VLA-v0 TraceShapeHard-VLA-v0 TraceShapeEasy-VLA-v0 TraceShapeMedium-VLA-v0 BunchOfColors5-VLA-v0 SeqOfColors5-VLA-v0 ChainOfColors5-VLA-v0 TimedTransferEasy-Long-VLA-v0 RememberShapeAndColor3x3-Long-VLA-v0 GatherAndRecall3-VLA-v0 RememberShapeAndColor3x2-Long-VLA-v0 BunchOfColors7-VLA-v0 RememberShapeAndColor5x3-Long-VLA-v0 RememberShape3-Long-VLA-v0 ShellGameShuffleColorLampTouch-Long-VLA-v0 ShellGameShuffleTouch-Long-VLA-v0 RememberColor3-Long-VLA-v0 RememberShape9-Long-VLA-v0 SeqOfColors7-VLA-v0 ChainOfColors7-VLA-v0 RememberShape5-Long-VLA-v0 BatteriesCheckerEasy-3-VLA-v0 RememberColor9-Long-VLA-v0 RememberColor5-Long-VLA-v0 BunchOfColors3-Long-VLA-v0 TimedTransferMedium-Long-VLA-v0 BlinkCountButtonPressMedium-Long-VLA-v0 BunchOfColors5-Long-VLA-v0 GatherAndRecall5-VLA-v0 ChainOfColors3-Long-VLA-v0 SeqOfColors3-Long-VLA-v0 BunchOfColors7-Long-VLA-v0 TraceShapeSeqEasy-VLA-v0 BatteriesCheckerHard-3-VLA-v0 TraceShapeSeqHard-VLA-v0 TraceShapeSeqMedium-VLA-v0 GatherAndRecall7-VLA-v0 SeqOfColors5-Long-VLA-v0 ChainOfColors5-Long-VLA-v0 BatteriesCheckerEasy-6-VLA-v0 BlinkCountButtonPressHard-Long-VLA-v0 GatherAndRecall9-VLA-v0 SeqOfColors7-Long-VLA-v0 ChainOfColors7-Long-VLA-v0 TimedTransferHard-Long-VLA-v0 BatteriesCheckerHard-6-VLA-v0 Dataset scale by task (90 tasks) Object Spatial Capacity Temporal Negative Sequential Procedural Prospective Tracking Checklist Figure 29: Total released frames per task, sorted descending, colored by memory type. The same color maps to the same memory type in every figure in this paper (Table 1). Numeric values for every task are in Table 33 below. 44 Table 33: Full per-task statistics for all 90 canonical tasks. Every task con- tributes 250 episodes. Lengths, horizons, and gaps are in environment steps. In theGap column a bare number is a constant the environment sets, median (p25–p75)is that constant drawn afresh each episode, undetermined marks an interval that ends when the agent reaches some state rather than after a stated number of steps,and cue never hidden marks a task whose source states no interval exists. Env ID Memory type Split Horizon Source Mean len. Median len. Oracle succ. Gap ShellGameTouch-VLA-v0 Spatial Short 30 PPO 21.5 22 1.000 0 ShellGamePush-VLA-v0 Spatial Short 30 PPO 15.0 15 1.000 0 InterceptSlow-VLA-v0 Spatial Short 60 PPO 43.8 43 1.000 cue never hidden InterceptMedium-VLA-v0 Spatial Short 60 PPO 41.8 41 1.000 cue never hidden InterceptFast-VLA-v0 Spatial Short 60 PPO 31.6 30 1.000 cue never hidden InterceptGrabSlow-VLA-v0 Spatial Short 60 PPO 27.6 24 1.000 cue never hidden InterceptGrabMedium-VLA-v0 Spatial Short 60 PPO 34.7 33 1.000 cue never hidden InterceptGrabFast-VLA-v0 Spatial Short 60 PPO 49.8 50 1.000 cue never hidden RotateLenientPos-VLA-v0 Spatial Short 60 PPO 29.3 29 1.000 cue never hidden RotateLenientPosNeg-VLA-v0 Spatial Short 60 PPO 19.8 19 1.000 cue never hidden RotateStrictPos-VLA-v0 Spatial Short 90 PPO 34.4 31 1.000 cue never hidden RotateStrictPosNeg-VLA-v0 Spatial Short 90 PPO 24.7 22 1.000 cue never hidden TakeItBack-VLA-v0 Spatial Short 60 PPO 28.9 27 1.000 undetermined RememberColor3-VLA-v0 Object Short 25 PPO 15.4 15 1.000 3 (2–4) RememberColor5-VLA-v0 Object Short 25 PPO 15.3 15 1.000 3 (2–4) RememberColor9-VLA-v0 Object Short 25 PPO 15.1 15 1.000 3 (2–4) RememberShape3-VLA-v0 Object Short 25 PPO 14.9 15 1.000 3 (2–4) RememberShape5-VLA-v0 Object Short 25 PPO 15.1 15 1.000 3 (2–4) RememberShape9-VLA-v0 Object Short 25 PPO 15.4 15 1.000 3 (2–4) RememberShapeAndColor3x2-VLA-v0 Object Short 25 PPO 15.0 15 1.000 3 (2–4) RememberShapeAndColor3x3-VLA-v0 Object Short 25 PPO 14.3 14 1.000 3 (2–4) RememberShapeAndColor5x3-VLA-v0 Object Short 25 PPO 15.4 15 1.000 3 (2–4) BunchOfColors3-VLA-v0 Capacity Medium 400 MP 161.8 158 1.000 3 (2–4) BunchOfColors5-VLA-v0 Capacity Medium 400 MP 241.4 242 1.000 3 (2–4) BunchOfColors7-VLA-v0 Capacity Medium 400 MP 325.3 327 1.000 3 (2–4) SeqOfColors3-VLA-v0 Capacity Medium 400 MP 165.6 165 1.000 3 (2–4) SeqOfColors5-VLA-v0 Capacity Medium 400 MP 252.9 253 1.000 3 (2–4) SeqOfColors7-VLA-v0 Capacity Medium 400 MP 341.3 342 1.000 3 (2–4) ChainOfColors3-VLA-v0 Sequential Medium 400 MP 165.4 165 1.000 3 (2–4) ChainOfColors5-VLA-v0 Sequential Medium 400 MP 253.1 253 1.000 3 (2–4) 45 Env ID Memory type Split Horizon Source Mean len. Median len. Oracle succ. Gap ChainOfColors7-VLA-v0 Sequential Medium 400 MP 341.4 342 1.000 3 (2–4) ShellGameShuffleTouch-VLA-v0 Tracking Short 60 PPO 42.1 42 1.000 28 (24–31) ShellGameShuffleColorLampTouch-VLA-v0 Tracking Short 60 PPO 43.6 44 1.000 28 (24–31) ShellGameColorLampTouch-VLA-v0 Spatial Short 30 PPO 15.4 15 1.000 0 FindImposterColor3-VLA-v0 Negative Short 25 PPO 15.3 15 1.000 3 (2–4) FindImposterColor5-VLA-v0 Negative Short 25 PPO 14.9 15 1.000 3 (2–4) FindImposterColor9-VLA-v0 Negative Short 25 PPO 15.6 16 1.000 3 (2–4) FindImposterShape3-VLA-v0 Negative Short 25 PPO 14.3 14 1.000 3 (2–4) FindImposterShape5-VLA-v0 Negative Short 25 PPO 15.4 15 1.000 3 (2–4) FindImposterShape9-VLA-v0 Negative Short 25 PPO 15.2 15 1.000 3 (2–4) FindImposterShapeAndColor3x2-VLA-v0 Negative Short 25 PPO 15.7 16 1.000 3 (2–4) FindImposterShapeAndColor3x3-VLA-v0 Negative Short 25 PPO 15.2 15 1.000 3 (2–4) FindImposterShapeAndColor5x3-VLA-v0 Negative Short 40 PPO 15.7 15 1.000 3 (2–4) BatteriesCheckerEasy-3-VLA-v0 Checklist Medium 540 MP 365.1 363 1.000 undetermined BatteriesCheckerEasy-6-VLA-v0 Checklist Long 1080 MP 728.4 727 1.000 undetermined BatteriesCheckerHard-3-VLA-v0 Checklist Long 1080 MP 705.6 699 1.000 undetermined BatteriesCheckerHard-6-VLA-v0 Checklist Long 2160 MP 1420.9 1416 1.000 undetermined BlinkCountButtonPressEasy-VLA-v0 Temporal Short 150 MP 94.6 94 1.000 0 BlinkCountButtonPressMedium-VLA-v0 Temporal Short 200 MP 121.3 121 1.000 0 BlinkCountButtonPressHard-VLA-v0 Temporal Medium 300 MP 150.3 153 1.000 0 RememberColor3-Long-VLA-v0 Object Medium 600 MP 334.1 337 1.000 250 (150–350) RememberColor5-Long-VLA-v0 Object Medium 600 MP 386.5 390 1.000 250 (150–350) RememberColor9-Long-VLA-v0 Object Medium 600 MP 379.9 378 1.000 250 (150–350) RememberShape3-Long-VLA-v0 Object Medium 600 MP 333.2 337 1.000 250 (150–350) RememberShape5-Long-VLA-v0 Object Medium 600 MP 342.1 349 1.000 250 (150–350) RememberShape9-Long-VLA-v0 Object Medium 600 MP 334.6 328 1.000 250 (150–350) RememberShapeAndColor3x2-Long-VLA-v0 Object Medium 600 MP 319.4 312 1.000 250 (150–350) RememberShapeAndColor3x3-Long-VLA-v0 Object Medium 600 MP 315.1 321 1.000 250 (150–350) RememberShapeAndColor5x3-Long-VLA-v0 Object Medium 600 MP 329.4 322 1.000 250 (150–350) BunchOfColors3-Long-VLA-v0 Capacity Long 700 MP 439.1 449 1.000 225 (138–313) BunchOfColors5-Long-VLA-v0 Capacity Long 700 MP 513.9 524 1.000 225 (138–313) BunchOfColors7-Long-VLA-v0 Capacity Long 700 MP 566.7 569 1.000 225 (138–313) SeqOfColors3-Long-VLA-v0 Capacity Long 800 MP 543.6 559 1.000 225 (138–313) SeqOfColors5-Long-VLA-v0 Capacity Long 1000 MP 724.0 738 1.000 225 (138–313) SeqOfColors7-Long-VLA-v0 Capacity Long 1200 MP 901.5 916 1.000 225 (138–313) ChainOfColors3-Long-VLA-v0 Sequential Long 800 MP 543.5 559 1.000 225 (138–313) ChainOfColors5-Long-VLA-v0 Sequential Long 1000 MP 725.2 740 1.000 225 (138–313) 46 Env ID Memory type Split Horizon Source Mean len. Median len. Oracle succ. Gap ChainOfColors7-Long-VLA-v0 Sequential Long 1200 MP 902.9 916 1.000 225 (138–313) ShellGameShuffleTouch-Long-VLA-v0 Tracking Medium 600 MP 333.5 337 1.000 250 (175–325) ShellGameShuffleColorLampTouch-Long-VLA-v0 Tracking Medium 600 MP 333.3 331 1.000 250 (175–325) BlinkCountButtonPressEasy-Long-VLA-v0 Temporal Long 1200 MP 205.7 199 1.000 0 BlinkCountButtonPressMedium-Long-VLA-v0 Temporal Long 1200 MP 491.2 490 1.000 0 BlinkCountButtonPressHard-Long-VLA-v0 Temporal Long 1200 MP 784.9 788 1.000 0 TraceShapeEasy-VLA-v0 Procedural Medium 250 MP 208.3 209 1.000 0 TraceShapeMedium-VLA-v0 Procedural Medium 300 MP 210.7 210 1.000 0 TraceShapeHard-VLA-v0 Procedural Medium 350 MP 207.5 207 1.000 0 TraceShapeSeqEasy-VLA-v0 Procedural Long 1500 MP 698.6 702 1.000 0 TraceShapeSeqMedium-VLA-v0 Procedural Long 1500 MP 718.6 720 1.000 0 TraceShapeSeqHard-VLA-v0 Procedural Long 1500 MP 714.9 706 1.000 0 TimedTransferEasy-VLA-v0 Temporal Short 200 MP 105.5 106 1.000 100 TimedTransferMedium-VLA-v0 Temporal Medium 250 MP 152.3 152 1.000 150 TimedTransferHard-VLA-v0 Temporal Medium 300 MP 203.2 203 1.000 200 TimedTransferEasy-Long-VLA-v0 Temporal Medium 600 MP 298.1 298 1.000 300 TimedTransferMedium-Long-VLA-v0 Temporal Long 900 MP 487.9 488 1.000 500 TimedTransferHard-Long-VLA-v0 Temporal Long 1200 MP 961.6 962 1.000 1000 GatherAndRecall1-VLA-v0 Prospective Short 200 MP 115.8 115 1.000 undetermined GatherAndRecall3-VLA-v0 Prospective Medium 400 MP 317.4 318 1.000 undetermined GatherAndRecall5-VLA-v0 Prospective Medium 600 MP 514.1 514 1.000 undetermined GatherAndRecall7-VLA-v0 Prospective Long 800 MP 720.9 721 1.000 undetermined GatherAndRecall9-VLA-v0 Prospective Long 1000 MP 898.1 900 1.000 undetermined 47 H Reference Baseline: Full Configuration Section 5.2 gives the reference baseline’s setup at the level needed to interpret its results. This appendix records the available training configuration and its provenance limits. Table 34: Reference π 0.5 baseline training configuration. FieldValue Initialization pi05_base cross-embodiment checkpoint, fully fine-tuned Codebase openpi-comet [Bai et al., 2025], a JAX/Flax fork of openpi Training dataLeRobot format, 14 tasks× 250 episodes (3,500 episodes, 798,296 frames) Image inputTwo cameras stacked, 128× 128× 3 each, upscaled to 224× 224 Action space pd_e_delta_pose, 7 active dims, padded to 32 Action chunk8 steps, executed fully open-loop Frame historyNone – policy sees only the current observation ProprioceptionDiscretized, injected into the text prompt (discrete_state_input=True) Global batch128, across 8 GPUs (fsdp=8) Peak learning rateNot recoverable from the source training project OptimizerNot recoverable from the source training project Checkpoint evaluatedStep 139,999 The training and evaluation lanes use distinct simulator seed ranges. Training episodes and the 20- per-task evaluation episodes (base_seed 10,000) therefore use distinct recorded seeds, although a seed alone does not establish that two simulator versions produce distinct initial states. I Comparison to Existing Benchmarks Table 35 compares dedicated simulation suites for memory-dependent manipulation on the axes Section 2 discusses in prose: scale, controlled variation of memory difficulty, and environment and learning support. Column definitions and the symbol legend appear in the note below the table. ScaleControlled memory difficulty Environment & learning BenchmarkBackendTasks Types Demos Ret. LoadH-strat.Dense rew.GPU vec. MemoryBench [Fang et al., 2025]RLBench / Coppelia32300✗ LIBERO-Mem [Chung et al., 2026]LIBERO / MuJoCo1041,000 ◦ ◦✗ RMBench [Chen et al., 2026]RoboTwin / SAPIEN9–450✗ ◦✗ RoboMME [Dai et al., 2026]ManiSkill / SAPIEN1641,600✗ RoboMemArena [Lei et al., 2026]LIBERO / MuJoCo2642,600✗ RoboTwin-MeM [Yang et al., 2026] RoboTwin / SAPIEN8–400 † ✗✓✗ MEMOBench [Sun et al., 2026]LIBERO / MuJoCo3041,500✗ MIKASA-Robo-VLA (ours)ManiSkill / SAPIEN901022,500✓ Table 35: Comparison of dedicated simulation benchmarks for memory-dependent robotic manipulation. Types follows each benchmark’s own taxonomy, not comparable across suites. Ret.: controlled retention-interval variation. Load: controlled variation of information amount or complexity. H-strat.: canonical evaluation across horizon regimes. Dense rew.: documented suite-wide dense rewards. GPU vec.: a released benchmark-level API for GPU-parallel execution.✓: explicit support.◦: partial or entangled support.✗: not documented or provided suite-wide. † RoboTwin-MeM’s 50 episodes per task (of eight) are shown for scale, not as an equivalent to other benchmarks’ demonstrations. J Supplementary External Evaluations This appendix collects three reference results that support Section 5.3 and Section 3’s remark on customization. None of them is part of the canonical 90-task benchmark evaluation. Each table’s caption states its own protocol and episode count, and the three must not be read against each other or against Table 3 as a single leaderboard. 48 J.1 Memory architecture comparison Table 36 reproduces, at per-task resolution, the comparison whose aggregates are quoted in Section 5.3: OpenVLA-OFT with and without a 64-token recurrent memory module (μVLA), both fine-tuned on the same five tasks and evaluated zero-shot on 18 more, as reported by Cherepanov et al. [2026b]. The underlying evaluation ran on 2026-05-03 (no-memory arm) and 2026-06-02 (μVLA arm), under an earlier release protocol that predates the wrapper and diagnostic-key differences described in Section 6, so these numbers are not under the same protocol this paper’s own baseline uses. ShellGameTouch-VLA-v0’s no-memory evaluation did not complete – environment creation raised an error – and is marked N/A rather than a false zero. Table 36: Per-task success on the 23-task evaluation reported by Cherepanov et al. [2026b]: OpenVLA-OFT without a memory module (exec-8) versusμVLA, the same backbone with a 64-token recurrent memory (receding horizon, memory updated every step). 100 episodes per task, checkpoint step 150000. Trained tasks are marked † . The other 18 are zero-shot/held-out for both arms. Evaluated under an earlier release protocol (see Section 6). N/A marks an evaluation that did not complete and so is not scored. TaskNo memory μVLA InterceptFast0.000.29 InterceptGrabFast0.000.00 InterceptGrabMedium0.000.00 InterceptGrabSlow0.000.00 InterceptMedium † 0.360.55 InterceptSlow0.050.08 RememberColor30.000.91 RememberColor5 † 0.040.94 RememberColor90.030.38 RememberShape30.000.39 RememberShape50.000.42 RememberShape90.000.26 RememberShapeAndColor3x20.030.62 RememberShapeAndColor3x3 † 0.030.84 RememberShapeAndColor5x30.010.27 RotateLenientPos0.060.01 RotateLenientPosNeg0.030.03 RotateStrictPos0.000.00 RotateStrictPosNeg0.000.00 ShellGamePick0.000.00 ShellGamePush † 0.830.96 ShellGameTouchN/A0.00 TakeItBack † 0.830.98 Mean (5 trained tasks)0.4180.854 J.2 Execution-regime comparison Table 37 reports two independently trained stockπ 0.5 (LeRobot port,lerobot/pi05_base) check- points on five tasks, each evaluated twice: once under the open-loop 8-action chunk this paper’s own baseline uses, once under one-action-per-inference execution with the same weights. Both checkpoints trained for 30,000 steps, global batch 64, chunk size 8. Loader A shuffles episodes, and loader B preserves episode order. Table 37: Referenceπ 0.5 (LeRobot port), two independently trained checkpoints, each evaluated under two execution regimes at inference: exec-8 (open-loop chunk of 8 actions) and exec-1 (receding horizon, one action per inference). 100 episodes per task per cell. Not part of the canonical 90-task benchmark evaluation. LoaderRegime ShellGamePush InterceptMedium TakeItBack RememberColor5 RSC3x3 Mean A (shuffled) exec-80.930.300.680.140.07 0.424 A (shuffled) exec-10.280.420.770.280.14 0.378 B (episodic) exec-80.950.310.770.180.03 0.448 B (episodic) exec-10.330.510.810.180.15 0.396 49 J.3 Benchmark customization case study Table 38 illustrates that the benchmark’s task parameters are meant to be re-specified, not only con- sumed as released. Five*-Medium-VLA-v0variants were created for a single test:μVLA with its re- current memory updated once per action chunk rather than every step, compared across two truncated- backpropagation-through-time (TBPTT) window lengths (K = 2, 16 environment steps, andK = 8, 64 steps) against a retention interval of 37–40 steps onRememberColor5-Medium-VLA-v0. These five variants are not part of the canonical 90-task suite and are excluded from every count, table, and figure elsewhere in this paper. They exist only as this worked example. The window that covers the retention interval (K = 8) recovers 0.540 success on that task against 0.080 for the window that does not (K = 2), matching the customization’s own design intent. Neither arm’s training run reached its full step budget before being halted by the user, so the aggregate means in the table are not a matched final comparison. Table 38: Benchmark-customization case study:μVLA with memory updated once per 8-action chunk (full open loop), trained on 3 canonical tasks plus 5 custom *-Medium-VLA-v0 variants created for this test and excluded from every count of the canonical 90-task suite.Kis the TBPTT window in chunks (env-steps = 8K). 50 episodes per task. TaskK=2 (16 steps, step 140,000)K=8 (64 steps, best step 42,500) BunchOfColors30.000.00 ChainOfColors30.000.00 GatherAndRecall30.240.26 RememberColor3-Medium0.320.52 RememberColor5-Medium0.080.54 RememberShapeAndColor3x2-Medium0.180.44 RememberShapeAndColor3x3-Medium0.060.32 ShellGameColorLampTouch-Medium0.300.32 Mean (8 tasks)0.1470.300 K Rollout Contact Sheet Figures 30–39 tile a filmstrip for every one of the 90 tasks, one sheet per memory type, as a visual index into Appendix F’s per-family detail. These frames are the render channel described there, so they carry debug text the released observations do not. A single sheet of all 90 tasks is taller than a page, so it prints at a fraction of its natural size and its task labels become unreadable. One sheet per memory type fits the page’s width instead and prints at full size. 50 Object (18) RememberColor3-Long RememberColor3 RememberColor5-Long RememberColor5 RememberColor9-Long RememberColor9 RememberShape3-Long RememberShape3 RememberShape5-Long RememberShape5 RememberShape9-Long RememberShape9 RememberShapeAndColor3x2-Long RememberShapeAndColor3x2 RememberShapeAndColor3x3-Long RememberShapeAndColor3x3 RememberShapeAndColor5x3-Long RememberShapeAndColor5x3 Figure 30: Rollout filmstrips for the 18 Object tasks, one row per task, 8 evenly spaced frames per rollout, left to right. The header color is this memory type’s color throughout the paper (Table 1). 51 Spatial (14) InterceptFast InterceptGrabFast InterceptGrabMedium InterceptGrabSlow InterceptMedium InterceptSlow RotateLenientPos RotateLenientPosNeg RotateStrictPos RotateStrictPosNeg ShellGameColorLampTouch ShellGamePush ShellGameTouch TakeItBack Figure 31: Rollout filmstrips for the 14 Spatial tasks, one row per task, 8 evenly spaced frames per rollout, left to right. The header color is this memory type’s color throughout the paper (Table 1). 52 Capacity (12) BunchOfColors3-Long BunchOfColors3 BunchOfColors5-Long BunchOfColors5 BunchOfColors7-Long BunchOfColors7 SeqOfColors3-Long SeqOfColors3 SeqOfColors5-Long SeqOfColors5 SeqOfColors7-Long SeqOfColors7 Figure 32: Rollout filmstrips for the 12 Capacity tasks, one row per task, 8 evenly spaced frames per rollout, left to right. The header color is this memory type’s color throughout the paper (Table 1). 53 Temporal (12) BlinkCountButtonPressEasy-Long BlinkCountButtonPressEasy BlinkCountButtonPressHard-Long BlinkCountButtonPressHard BlinkCountButtonPressMedium-Long BlinkCountButtonPressMedium TimedTransferEasy-Long TimedTransferEasy TimedTransferHard-Long TimedTransferHard TimedTransferMedium-Long TimedTransferMedium Figure 33: Rollout filmstrips for the 12 Temporal tasks, one row per task, 8 evenly spaced frames per rollout, left to right. The header color is this memory type’s color throughout the paper (Table 1). 54 Negative (9) FindImposterColor3 FindImposterColor5 FindImposterColor9 FindImposterShape3 FindImposterShape5 FindImposterShape9 FindImposterShapeAndColor3x2 FindImposterShapeAndColor3x3 FindImposterShapeAndColor5x3 Figure 34: Rollout filmstrips for the 9 Negative tasks, one row per task, 8 evenly spaced frames per rollout, left to right. The header color is this memory type’s color throughout the paper (Table 1). Sequential (6) ChainOfColors3-Long ChainOfColors3 ChainOfColors5-Long ChainOfColors5 ChainOfColors7-Long ChainOfColors7 Figure 35: Rollout filmstrips for the 6 Sequential tasks, one row per task, 8 evenly spaced frames per rollout, left to right. The header color is this memory type’s color throughout the paper (Table 1). 55 Procedural (6) TraceShapeEasy TraceShapeHard TraceShapeMedium TraceShapeSeqEasy TraceShapeSeqHard TraceShapeSeqMedium Figure 36: Rollout filmstrips for the 6 Procedural tasks, one row per task, 8 evenly spaced frames per rollout, left to right. The header color is this memory type’s color throughout the paper (Table 1). Prospective (5) GatherAndRecall1 GatherAndRecall3 GatherAndRecall5 GatherAndRecall7 GatherAndRecall9 Figure 37: Rollout filmstrips for the 5 Prospective tasks, one row per task, 8 evenly spaced frames per rollout, left to right. The header color is this memory type’s color throughout the paper (Table 1). Tracking (4) ShellGameShuffleColorLampTouch-Long ShellGameShuffleColorLampTouch ShellGameShuffleTouch-Long ShellGameShuffleTouch Figure 38: Rollout filmstrips for the 4 Tracking tasks, one row per task, 8 evenly spaced frames per rollout, left to right. The header color is this memory type’s color throughout the paper (Table 1). 56 Checklist (4) BatteriesCheckerEasy-3 BatteriesCheckerEasy-6 BatteriesCheckerHard-3 BatteriesCheckerHard-6 Figure 39: Rollout filmstrips for the 4 Checklist tasks, one row per task, 8 evenly spaced frames per rollout, left to right. The header color is this memory type’s color throughout the paper (Table 1). 57