Paper deep dive
ValueFormer: A Causal Transformer Value Function with Stage-Aware Labels for Semi-Autonomous Vision-Language-Action Policies
Inkyu Sa, Konstantin Stulov, Rajat Bhageria
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/5/2026, 4:43:58 AM
Summary
ValueFormer is a compact, policy-agnostic causal transformer that provides per-frame value signals for Vision-Language-Action (VLA) policies. It utilizes a frozen DINOv3 backbone and stage-aware labeling to produce a smooth Monte Carlo value for advantage estimation and a binary value for online mistake detection. Evaluated on a real-robot bimanual sandwich-assembly task, it improved task completion from 70% to 85% and reduced serving costs significantly.
Entities (8)
Relation Signals (6)
ValueFormer → evaluatedon → Sandwich-assembly task
confidence 95% · On a real-robot bimanual sandwich-assembly task (1,427 episodes)
ValueFormer → usesbackbone → DINOv3
confidence 95% · ValueFormer, a compact policy-agnostic causal transformer over a frozen DINOv3 backbone
ValueFormer → emitssignal → V_mc
confidence 92% · emits two per-frame signals in one forward pass: a smooth Monte Carlo value, V_mc
ValueFormer → emitssignal → V_bin
confidence 92% · emits two per-frame signals in one forward pass: ... a sharp binary value for online mistake detection
ValueFormer → improvesperformanceof → Sandwich-assembly task
confidence 90% · lifts task completion from 70% to 85%
ValueFormer → compatiblewith → π0 family
confidence 85% · policy-agnostic alternative: the same value head can supervise any VLA in the π0 family
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Vision-Language-Action (VLA) policies trained by behavior cloning fail silently: from the action stream alone, a collapsing rollout looks much like one making clean progress, because imitation supplies no notion of progress. Reinforcement learning would supply one, but it is impractical here, where real-robot experience is costly and deformable food resists simulation. The cheap alternative, a terminal success / failure bit, is learnable in principle yet far too sparse to say when a rollout went wrong. We argue that the per-frame label, not the architecture, is the hard part: to be useful it must be dense, continuous, and correctly shaped. We present ValueFormer, a compact policy-agnostic causal transformer over a frozen DINOv3 backbone that emits two per-frame signals in one forward pass: a smooth Monte Carlo value, V_mc, for advantage estimation and a sharp binary value for online mistake detection, targets that pull in opposite directions by design. Failed episodes are labeled with a stage-aware, success-then-decay return that preserves the success curve before the failure stage, and detection is supervised from mistake intervals rather than a single failure time, so mistakes the policy recovers from also carry signal. On a real-robot bimanual sandwich-assembly task 1,427 episodes), a critic-derived per-frame training weight lifts task completion from 70% to 85% (within noise at n=20), and a batched bf16 encoder cuts the live serving cost 3~5 times so the critic runs at 2 Hz alongside the policy on a single GPU.
Tags
Links
- Source: https://arxiv.org/abs/2608.02958v1
- Canonical: https://arxiv.org/abs/2608.02958v1
Trouble viewing inline? Open PDF directly →
Full Text
114,291 characters extracted from source content.
Expand or collapse full text
ValueFormer: A Causal Transformer Value Function with Stage-Aware Labels for Semi-Autonomous Vision-Language-Action Policies Inkyu Sa, Konstantin Stulov, and Rajat Bhageria Abstract Vision-Language-Action (VLA) policies trained by behavior cloning fail silently: from the action stream alone, a collapsing rollout looks much like one making clean progress, because imitation supplies no notion of progress. Reinforcement learning would supply one, but it is impractical here, where real-robot experience is costly and deformable food resists simulation. The cheap alternative, a terminal success / failure bit, is learnable in principle yet far too sparse to say when a rollout went wrong. We argue that the per-frame label, not the architecture, is the hard part: to be useful it must be dense, continuous, and correctly shaped. We present ValueFormer, a compact policy-agnostic causal transformer over a frozen DINOv3 backbone that emits two per-frame signals in one forward pass: a smooth Monte Carlo value VmcV_mc for advantage estimation and a sharp binary value VbinV_bin for online mistake detection, targets that pull in opposite directions by design. Failed episodes are labeled with a stage-aware, success-then-decay return that preserves the success curve before the failure stage, and detection is supervised from mistake intervals rather than a single failure time, so mistakes the policy recovers from also carry signal. On a real-robot bimanual sandwich-assembly task (1,4271,427 episodes), a critic-derived per-frame training weight lifts task completion from 70%70\% to 85%85\% (within noise at n=20n=20), and a batched bf16 encoder cuts the live serving cost 33–5×5× so the critic runs at 22 Hz alongside the policy on a single GPU. I Introduction Figure 1: ValueFormer running live alongside a VLA policy on a held-out rollout. From a single forward pass the dual head emits two complementary per-frame signals, overlaid on the camera views: a smooth progress value V(s)∈[0,1]V(s)∈[0,1] (green) and a sharp failure probability 1−Vbin1-V_bin (orange), thresholded at the red line. Top: before an external disturbance, a healthy frame (V=0.69V=0.69, failure probability 0.180.18), no alert. Bottom: after an external disturbance is applied to the lettuce, a mistake begins, the lettuce is skipped and the policy proceeds directly to the top bun. The failure probability saturates to 1.001.00 (V=0.61V=0.61) and raises an operator-facing value alert while the smooth value has barely moved. The sharp head thus supplies the actionable abort signal that the deliberately smooth V(s)V(s) does not. See Section VI-E. Bimanual manipulation in unstructured human environments, such as a restaurant sandwich station, is now within reach of modern Vision-Language-Action (VLA) policies. Models from the π0 _0 family [1, 2, 3], together with OpenVLA [4], RDT-1B [5], and GigaBrain [6], have reduced what used to be months of engineering to a few hours of demonstration. A robot that was recently a research artifact now performs real bimanual manipulation on a live station. What these policies typically cannot do is tell the operator how they are doing, especially when they have been trained only on successful episodes. For instance, a rollout that has stalled on a stubborn lettuce leaf, one that is about to place a second patty, and one that is making clean progress can all produce the same kind of output: a flow-matched action chunk, a video stream, and a stream of joint commands. The policy carries no calibrated notion of progress. For an operator who oversees a fleet of such robots, this gap can be the difference between a useful semi-autonomous system and one that demands a person at every screen. A natural fix is a value function: a second, lightweight model that watches the same observations the policy sees and returns a scalar tracking how close the episode is to task completion. Classical actor-critic methods give this for free when the policy is trained with reinforcement learning. Flow-matching VLAs typically do not enjoy this luxury; they are usually trained by behavior cloning on expert data, with no value baseline, no critic, and no explicit reward signal. Training the policy with reinforcement learning instead is not a practical answer in our setting: on-policy exploration on a real station is slow, wasteful of food, and unsafe to run unattended, and the obvious escape route of learning in simulation is blocked by the contents of the task, since deformable and granular food (a lettuce leaf that folds, a cheese slice that sticks, a patty that tips) is precisely what current simulators reproduce least faithfully. A value function must therefore be learned separately from the policy, and three problems make this harder than it looks. Why is a terminal outcome bit not enough? Before any of those, there is a prior question about what supervision to ask for at all. Every rollout already carries the cheapest label imaginable, one bit at the end of the episode saying whether it succeeded. That bit is not useless, and we want to be precise about this: it is learnable in principle, and with enough episodes an outcome classifier will separate good rollouts from bad ones. What it cannot do is say when. One bit spread over a two-minute, six-stage episode is the reward-sparsity problem that makes reinforcement learning expensive on real robots, reappearing here as a labeling problem, and it is a poor fit for either consumer of the signal. An advantage estimate needs to know whether the last few seconds moved the rollout closer to a finished sandwich or further away, which is a question about the difference between two nearby frames and is undefined when the only label sits at the terminal. A safety filter needs the frame at which a mistake began, so an operator has time to intervene, and an episode-level verdict arrives long after that moment has passed. Density is what makes the signal actionable, so we ask for a value at every control step instead. Density alone, however, is not sufficient, and this is the part that cost us the most effort. A dense but wrongly shaped label is worse than no label at all, because the network fits it faithfully: the textbook flat-zero target below is dense, and it actively teaches the model that ordinary good observations predict failure. What both consumers need is a target that is dense, continuous rather than binary, and shaped so that its level and its slope carry meaning. Getting that shape right, rather than scaling the encoder or the transformer, is the central contribution of this paper, and Section VI-D quantifies how much it matters: across five label shapes under one fixed architecture the spread in validation loss is roughly 4×4×, while no architectural change we tried moved the result comparably. What is the label? A natural starting point is a value label that rises monotonically toward 11 on successful episodes and sits at zero on failures. The textbook instantiation of that idea is a Monte Carlo (MC) return: V rises to V=1V=1 at success on successful episodes (discounting backward from the terminal frame) and is held flat at zero on failures. It seems reasonable but is badly miscalibrated. A failed rollout usually matches a successful one over its early stages, up to the point where the mistake occurs; forcing V≡0V≡ 0 over that prefix teaches the model that the observations themselves mean failure, which then bleeds into clean successes at deployment and produces noisy, zero-collapsing predictions on held-out rollouts (Section VI-D). When did the failure happen? Policy rollouts carry an episode-level success / failure flag and, at best, a one-line human comment (“two patties”, “scratching cheese”, “couldn’t pick lettuce”). Without a per-episode failure stage and time, stage-aware labels cannot be built. We tried a vision-language model (Gemma-4-26B) as an automated stage-completion labeler; it was not accurate enough to be the only source of truth, so this paper relies on manual annotation of the rollout failure set. What architecture actually works? A single-frame value is jittery because rollouts are inherently periodic: arms move in and out of view, grippers open and close. A long recurrent model is overkill and slow at deployment. What we want is a compact causal model that consumes a short, fixed window of history and emits one value per frame at policy rate. We address these three problems together. Our contribution is a compact pipeline that takes an episode set in the LeRobot v2 format [7], returns per-frame Monte Carlo value labels that are aware of both task stages and when failure occurred, and trains a causal transformer on frozen DINOv3 [8] features together with joint state and time features to predict V(s)∈[0,1]V(s)∈[0,1] at every control step. We call the resulting model ValueFormer: the model is a transformer over a short temporal window, and what it produces is a scalar value. On a real-robot sandwich-assembly task with 1,4271,427 episodes, ValueFormer reaches validation MSE ≈3×10−4≈ 3× 10^-4, cleanly separates success and failure trajectories, and reproduces four canonical rollout signatures: a smoothly ascending curve on clean successes, a characteristic dip and recovery on success-with-retry, a rise-then-drop on early collapse, and a flat-low profile on stuck-scratching episodes. The contributions of this paper are: • Stage-aware, success-then-decay labeling: pre-failure frames of a failed episode follow the success curve and only decay smoothly after the failure stage. The naive flat-zero alternative is actively harmful: it produces oscillatory predictions on clean successes. • A systematic ablation of four alternative fail-episode label shapes (outcome-scaled, cliff, α-linear mix, late-diverge), trained and evaluated under an identical ValueFormer recipe. The smooth-decay form we use is the only one that simultaneously avoids an outcome-dependent leak at t=0t=0 and a hard discontinuity at the failure frame, and it is also the strongest in terms of validation BCE loss and rollout MAE. • A dual-head architecture with shared backbone that emits a smooth VmcV_mc (advantage critic) and a sharp per-frame VbinV_bin (online mistake detection) from one 22-layer causal transformer. The binary head is supervised by segment-based mistake intervals (tstart,tend)(t_start,t_end), so transient mistakes that the policy later recovers from in otherwise-successful rollouts also contribute training signal, a regime that a single failure point cannot capture. • ValueFormer, a compact (≈3.5≈\!3.5 M-parameter) policy-agnostic causal transformer over a frozen DINOv3 ViT-L/16 with six-view encoding (three cameras and three gripper ROIs). On 1,4271,427 real-robot sandwich-assembly episodes it reproduces four held-out signatures (clean success, success-with-retry, early collapse, stuck-scratching) while leaving the underlying VLA unchanged. • Two deployment results. We profile ValueFormer’s live serving cost when it shares a GPU with the policy and give a batched-encoder ++ bf16 path that cuts the per-tick cost 33–5×5× (Section VI-H); and in an on-robot A/B we use the critic to supply a per-frame policy-training weight that improves subtask quality and removes a repeat-pick failure mode against a flag-only intervention baseline, on identical data (Section VI-G). One deployment-time use of V(s)V(s), a critic-derived per-frame weight for VLA post-training, is evaluated on-robot here (Section VI-G), and we profile and optimize the critic’s live serving cost (Section VI-H); the remaining uses (abort/retry gating, advantage conditioning, and offline trajectory filtering) are sketched as future work in Section VIII. I Related Work ValueFormer sits at the intersection of several active threads: the recent π0 _0-family of flow-matching VLAs, classical value / critic learning for manipulation, failure-detection systems for behavior-cloning policies, per-frame trajectory-supervision labeling schemes, and hierarchical / semi-autonomous control. We review each in turn and close with a critical comparison. I-A Vision-Language-Action Policies The VLA paradigm collapsed what used to be three separate modules (perception, language grounding and decision making, control) into a single foundation model. RT-1 [9] first showed that a Transformer trained on ∼130 \!130k real-robot demonstrations could generalize across tasks, embodiments, and environments. RT-2 [10] then demonstrated that fine-tuning a web-scale VLM as the backbone transfers broad semantic knowledge to the robot. OpenVLA [4] released a 7B-parameter open-source VLA that made this recipe widely reproducible, and OpenPI [11] released a matching open implementation of the π0 _0 stack, which is the basis of the policy we pair ValueFormer with here. The π0 _0 family is the line of work our paper most directly builds on. π0 _0 [1] introduced flow matching [12] as the action head, showing that continuous, high-frequency bimanual actions can be generated far more smoothly than with autoregressive discretization. π0.5 _0.5 [2] extended this to open-world household tasks with a fleet of mobile manipulators, adding co-training on heterogeneous data and an explicit high-level planner. π0.6∗ _0.6^* / RECAP [3] is the closest prior work in spirit: a post-training recipe (RL with Experience and Corrections via Advantage-conditioned Policies) in which a VLA bootstraps its own improvement from autonomous experience. RECAP trains a distributional value function that predicts steps-to-success on accumulated rollout data, binarizes the resulting advantage into a discrete good/bad (True/False) indicator, and retrains the VLA conditioned on that indicator; the value function and the policy are trained in alternating rounds (fit the value, compute advantages, fine-tune the policy, collect new rollouts, repeat), rather than jointly with a shared loss. ValueFormer is a policy-agnostic alternative: the same value head can supervise any VLA in the π0 _0 family without modifying the policy’s input format or training loop. Diffusion-based and hybrid variants round out the landscape. RDT-1B [5] scaled bimanual diffusion policies to 1.2B parameters; Diffusion Policy [13] established the action-diffusion recipe; ACT [14] introduced action chunking with a transformer; and Xiaomi Robotics-0 [15] released a recent open-source VLA optimized for real-time execution. GigaBrain-0.5M [6] moved in a complementary direction, using a learned world model to augment scarce real-robot data with synthetic rollouts. None of these systems ship with a per-frame progress signal that an operator can inspect at deployment time; the value function we describe is, to our knowledge, the first such signal that has been built for the π0 _0 family specifically. I-B Value, Critic, and Reward Models for Manipulation Classical actor-critic RL produces a value function as a by-product of training [16, 17, 18]. In behavior cloning, where no reward is ever supplied, this is not free, and a growing body of work studies how to learn values from demonstration or from experience. Q-Transformer [19] learned a scalable value head for high-capacity transformer policies in the Robotics Transformer family, but required reward-labeled data. IQL and CQL [20, 21] fit value functions in the offline RL setting without environment interaction; both assume a scalar reward per step. Value Implicit Pre-training (VIP) [22] and LIV [23] go one step further and learn goal-conditioned value representations from video alone, treating video progress as an implicit reward. Their values are calibrated in a latent embedding space rather than in a task-progress space a human supervisor can read off a screen, a distinction that matters for the operator-facing use case we target. GCBC [24] and GoFAR [25] show that goal-conditioned behavior cloning and IQL-style critics can be combined to rank trajectories for retrieval or relabeling. DPPO [26] fine-tunes diffusion policies with PPO, showing that on-policy RL is tractable on top of imitation-trained actors. The closest single work to ours is the distributional value function used inside π0.6∗ _0.6^* / RECAP [3], which is trained to predict steps-to-success on accumulated autonomous rollouts; the advantage is then thresholded into a binary good/bad indicator and the VLA is supervised-fine-tuned conditioned on that indicator. VLAC [27] takes a different route by unifying critic and actor in one InternVL-based model that emits both a progress delta and an action. Both works differ from ValueFormer in two respects: (i) their value head is refit alongside the VLA whenever new rollout data arrives, and (i) they use wall-clock-indexed targets rather than task-stage-indexed targets, which leaves the pre-failure prefix of a failed episode mislabeled. What distinguishes ValueFormer is the explicit treatment of when a failure occurred and of partial credit, together with a policy-agnostic design that runs alongside a frozen VLA. I-C Failure Detection for Behavior-Cloning Policies Failure detection on behavior-cloning rollouts has seen rapid progress in the past two years. SAFE [28] trains a small latent-feature detector on top of VLA internals with conformal calibration; AHA [29] fine-tunes a VLM to emit natural-language explanations of what went wrong; I-FailSense [30] scales detection across embodiments; and StepEval [31] replaces the binary success label with a per-subtask rubric. A separate line of work uses foundation models as automatic reward or success labelers: RoboCLIP [32] and LIV [23] use CLIP-style embeddings to score task completion, while Eureka [33] asks GPT-4 to write reward code directly. Most of these works operate at the episode level (“did it succeed?”) or emit a binary flag per subtask. This is too coarse for a deployment-time supervisor: a binary flag fires at the point of failure, at which moment it is already too late to intervene. ValueFormer takes the middle road. It outputs a smooth per-frame value V(s)∈[0,1]V(s)∈[0,1] at policy rate, which dips early on retries and recovers when a stage completes, and which is supervised by a stage-aware Monte Carlo target rather than by an automated foundation-model labeler. I-D Labeling Schemes for Per-Frame Trajectory Supervision Independently of the architecture, the choice of label fixes what the model can learn. Published work on long-horizon, multi-stage video offers five families of per-frame supervision, each with a different bias-variance and annotation-cost trade-off. (i) Per-frame multi-class action / phase labels. Surgical action recognition has converged on this scheme. Every frame carries a phase or gesture class, and some classes encode error gestures (e.g., JIGSAWS G15 Loose Suture) [34]. Annotation is performed at interval boundaries and expanded to per-frame for training. State-of-the-art systems on JIGSAWS, Cholec80 [35], and similar surgical benchmarks use temporal convolution networks (MS-TCN [36]) or transformers (ASFormer [37]) over frame embeddings. The same per-frame multi-class formulation is dominant for temporal action localization on THUMOS [38], ActivityNet [39], and EPIC-Kitchens [40], where ActionFormer [41] is a representative architecture. Mistakes are detected as frames whose argmax class is an error class, and motion is captured by the temporal model rather than by the per-frame label. (i) Per-step process-reward labels. In language-model reasoning, process reward models (PRMs) attach a binary correct / incorrect label to every step of a chain of thought, with the verifier supervised from human-annotated step-level judgments (PRM800K) [42]. This is the closest direct precedent in the literature for per-frame error supervision in a sequential decision process: the supervision shape is identical to interval-based binary mistake labels in robotics. (i) Goal-conditioned progress and distance. VIP [22], LIV [23], and R3M [43] learn a per-frame scalar progress signal from successful trajectories alone via contrastive temporal-distance objectives. No failure annotations are required, but mistakes can only be inferred indirectly, as frames where progress regresses or stalls. (iv) Sparse-reward Monte Carlo returns. Value functions trained against episode-level outcomes via discounted MC returns [44, 3] produce a smooth per-frame target. This is the family our default labels (Section IV) belong to. The known limitation is mode collapse on near-success failure cases, where the smooth target underweights short, locally critical late-trajectory mistakes. (v) VLM-as-judge and trajectory-contrastive labels. A separate line of recent work bypasses manual annotation entirely: foundation models score each frame (RoboCLIP [32], AHA [29], and reward-code synthesis approaches such as Eureka [33]), or success / failure trajectories are paired contrastively so that mistakes surface as low-similarity-to-success-prototype frames. We treat these as alternatives to manual labels and discuss them in Section I-C. Implications for ValueFormer. ValueFormer adopts family (iv) by default, with the stage-aware modification described in Section IV that recovers the partial-credit information family (iv) typically loses. The labeling-scheme ablation in Section V is internal to family (iv): it varies the post-failure decay shape while holding the architecture, dataset, and training recipe fixed. A cross-family comparison (per-frame multi-class with explicit error classes, PRM-style binary error labels inside annotated mistake intervals, and contrastive goal-distance) is left to future work; we expect each to capture a different slice of “mistake”, and a deployment-grade supervisor may eventually combine them. I-E Hierarchical and Semi-Autonomous Control A run-time value signal is, in practice, a hierarchical / semi-autonomous control hook. Hi Robot [45] introduced a two-level architecture in which a VLM supervises a low-level VLA and intervenes in natural language when execution deviates from the intended plan. SayCan [46] and Inner Monologue [47] pioneered the split for long-horizon tasks. Interactive Language [48] and HULC [49] demonstrated operator-in-the-loop corrections. The classical precedent is HG-DAgger [50], which gates human intervention with a learned uncertainty metric; ValueFormer is the VLA-era successor, replacing uncertainty with a calibrated progress value. None of these systems runs a calibrated per-frame progress signal for the low-level policy; everything is routed through a language-level supervisor. ValueFormer offers that supervisor (human or VLM) a numeric handle that can be thresholded or integrated cheaply, which we expect to reduce VLM query load in deployment. I-F Positioning Compared to prior value / critic learning for behavior cloning [22, 23, 19, 3, 27], ValueFormer differs in that it (i) explicitly models failure timing and partial credit through stage-aware labels, and (i) is policy-agnostic and runs alongside any π0 _0-family VLA without re-training the policy. A practical consequence is that V(s)∈[0,1]V(s)∈[0,1] is strictly more expressive than a binarized good/bad advantage indicator: it can be thresholded to reproduce RECAP’s discrete conditioning signal, and additionally supports recipes that a binary label cannot (AWR-style weighted behavior cloning, continuous-threshold operator gating, and calibrated offline trajectory ranking). Compared to failure-detection systems [28, 29, 30, 31], it returns a smooth continuous value rather than a binary flag, which allows it to dip early during retries and to support advantage estimation. Compared to hierarchical control [45, 46, 50], it provides the numeric progress signal that such systems currently lack. I Problem Formulation I-A Setting We work in the standard behavior-cloning VLA setting. A rollout is a sequence of observation-state-action tuples indexed by policy-rate timestep t∈0,…,T−1t∈\0,…,T-1\, τ=((t,t,t))t=0T−1,τ= ((o_t,q_t,a_t) )_t=0^T-1~, (1) where to_t is a multi-view RGB observation (three cameras: top, left wrist, right wrist, plus optional ROI crops of the top-view and wrist-gripper regions), t∈ℝ14q_t ^14 is the bimanual joint-state vector (77 degrees of freedom per arm), and t∈ℝ14a_t ^14 is the action commanded by a fixed VLA policy πϕ _φ. Each rollout carries an episode-level outcome y∈0,1y∈\0,1\ (success / failure) and, for failures, an optional scalar failure time tfail∈[0,T)t_fail∈[0,T). At a coarser granularity, the task decomposes into NsN_s ordered stages =(s1,…,sNs)S=(s_1,…,s_N_s); for the sandwich-assembly task we use throughout, Ns=6N_s=6 with =(bottom_bun,patty,cheese,tomato,lettuce,top_bun)S=(bottom\_bun,patty,cheese,tomato,lettuce,top\_bun). I-B Goal We want a function Vθ:×→[0,1]V_θ:O×Q→[0,1] that, given the most recent window of frames of observation and state, returns a scalar progress value with the following semantics: V(s)≈1V(s)≈ 1 when the task is complete, V(s)V(s) rising while the rollout is making progress, V(s)V(s) flat while the rollout is stuck or retrying, and V(s)≈0V(s)≈ 0 on collapse. The model is not a reward function, because no environment reward is ever observed; it is a regressor onto a carefully chosen Monte Carlo label (Section IV-A). I-C Why the Naive MC Label Fails One possible, naive choice is Vnaive(t)=γT−1−t 1y=1V_naive(t)=γ^T-1-t\,1\y=1\ with γ∈(0,1)γ∈(0,1): positive on successes, zero on failures. The hidden assumption is that the observations on a failed episode were indicative of failure at every t. This holds only in the rare case of a rollout that goes wrong from its very first frame; for the great majority of failures it does not. A rollout typically fails partway through, and until that point its observations match those of a clean success, so its pre-failure frames carry no visible cue of the eventual outcome. Training on V≡0V≡ 0 over that prefix teaches the value network that these observations themselves are failures, a mistake that bleeds into clean-success inference at deployment: clean validation successes receive V values that oscillate near zero instead of tracking the success curve, even though nothing is wrong with the rollout (Section VI-D). The fix is therefore in the label, not the architecture: we redesign the per-frame supervision target rather than change the model. IV ValueFormer The three parts of the method are described in order of data flow: label generation first, then the ValueFormer architecture, and finally the training objective. Figure 2 gives the end-to-end view. Figure 2: ValueFormer system overview covering dataset construction, training, and real-time inference. Details in Sections IV-A–IV-D. IV-A Stage-Aware, Success-then-Decay Labels Per-frame scalar labels vk∈[0,1]v_k∈[0,1] are generated from an episode’s outcome y and, for failures, a stage-aware failure timing tfailt_fail. Let γ∈(0,1)γ∈(0,1) be the horizon discount, fixed to γ=0.99γ=0.99 in all experiments. Let k∈0,…,N−1k∈\0,…,N-1\ be the per-frame sample index after the 15×15× temporal subsampling of Section V-B, and N the number of sampled frames in the episode. Successful episodes. Labels follow a rising MC curve, vksucc=γ(N−1−k),k∈0,…,N−1,v_k^succ=γ^(N-1-k)~, k∈\0,…,N-1\~, (2) so that vN−1=1v_N-1=1 and v0=γN−1≈0.23v_0=γ^N-1≈ 0.23 for a typical ∼73 \!73-second episode at N=147N=147 samples. Failed episodes. Let sfail∈1,…,Nss_fail∈\1,…,N_s\ be the number of stages completed before the failure (determined as in Section IV-B), and let kfail=⌊(sfail/Ns)⋅N⌋k_fail= (s_fail/N_s)· N be its frame index. The label is vkfail=γ(N−1−k)k≤kfail,vkfail⋅γ(k−kfail)k>kfail.v_k^fail= casesγ^(N-1-k)&k≤ k_fail~,\\[2.0pt] v_k_fail·γ^(k-k_fail)&k>k_fail~. cases (3) Eq. (3) carries the whole contribution on the label side. The pre-failure prefix of a failed episode reuses the success curve of Eq. (2) because, up to kfailk_fail, the observations are those of a working rollout. After kfailk_fail, the label decays with the same γ. This gives a smooth differentiable target instead of a cliff, and encodes the intuition that a failure discovered late still deserves partial credit for the stages that were completed. Figure 3: Monte Carlo return labels. (Left) effect of the discount γ; (Center) success curve; (Right) stage-aware failure curve compared to the naive V≡0V≡ 0 baseline. Figure 4: Why the failure label is stage-aware, for a six-stage sandwich task (Section IV-A). Left: the naive alternative, where the success label rises as γN−1−kγ^N-1-k but every frame of a failed episode is flattened to V≡0V≡ 0, discarding all the progress the rollout made before its mistake. Right: the MC-smooth label used throughout, where a failed episode follows the success curve up to its failure stage and only then decays at rate γ. Failures at later stages therefore retain more of the success curve and peak higher (patty 0.330.33, cheese 0.410.41, lettuce 0.640.64), which is the partial credit the flat-zero label cannot express. Figure 5: The four alternative fail-episode label shapes (A: outcome-scaled, B: cliff, C-linear: α-mix, C-late: late-diverge) compared against MC-smooth (the curve we use; identical to the right panel of Figure 4). Each panel overlays three failure stages (patty / cheese / lettuce) and the success reference (dashed). A leaks the outcome at t=0t=0; B introduces a hard discontinuity at kfailk_fail; C-linear and C-late preserve the success start but distort the pre-failure curve over a long window. See Section V. The right panel of Figure 3 compares the flat-zero failure label against the stage-aware scheme of Eq. (3), and its left panel ablates the discount: γ=0.95γ=0.95 is too steep and the value rises only in the final few seconds; γ=0.999γ=0.999 is too flat, already starting near 11; γ=0.99γ=0.99 gives the curve we use in practice. Why exponential, not linear? A linear label vk=k/(N−1)v_k=k/(N-1) would also rise monotonically from 0 to 11, but it collapses the downstream advantage signal. The advantage estimator At=Vmc(t+HA)−Vmc(t)A_t=V_mc(t+H_A)-V_mc(t), for a lookahead horizon HAH_A, used in Section VI-F reduces under a linear label to the constant HA/(N−1)H_A/(N-1), independent of t: every frame of a successful rollout carries the same advantage, so a post-training filter cannot distinguish real progress from coasting. The exponential form keeps AtA_t non-constant and larger near the terminal, which matches the intuition that closing the last fraction of the distance to success is more informative than motion early in the rollout. A second, sharp target for online detection. The MC label of Eq. (3) is deliberately smooth: it changes by at most a factor of γ=0.99γ=0.99 from one 0.50.5 s sample to the next, which is what a critic wants but is the opposite of what a safety filter wants. An operator who needs an actionable abort signal does not want to wait for a decaying tail; they want a step at the moment a mistake begins. We therefore supplement VmcV_mc with a second per-frame target Vbin∈0,1V_bin∈\0,1\ built from segment-based mistake intervals rather than a single failure point. Let ℐe=(tstart(j),tend(j))j=1MeI_e=\(t^(j)_start,t^(j)_end)\_j=1^M_e be the set of annotated mistake intervals on episode e (Me=0M_e=0 if none), and let tkt_k be the wall-clock time of sample k. The label is vkbin=0∃jwithtstart(j)≤tk<tend(j),1otherwise.v^bin_k= cases0&∃\,j\;with\;t^(j)_start≤ t_k<t^(j)_end~,\\ 1&otherwise~. cases (4) Eq. (4) carries two design choices. First, vkbinv^bin_k is defined per frame from intervals, not from an episode-level success / failure flag, so an episode that succeeds overall can still contribute 0-labeled frames over a transient mistake (a missed grasp that the policy retries and recovers, a brief scratch on the cheese bin). The MC target of Eq. (3) cannot express this: a successful episode is labeled vksuccv^succ_k throughout regardless of how messy the path to the goal was. Second, the interval representation (tstart,tend)(t_start,t_end) supersedes the single-point failure time tfailt_fail used for the MC label. The MC head needs only to know when the rollout passed the point of no return, but the binary head needs to know which frames look like a mistake, and a transient mistake has a beginning and an end. The interval annotation schema we use (Section IV-B) allows up to three mistake intervals per episode plus an optional severity weight, which is multiplicatively applied to VmcV_mc over the interval so that recovered mistakes also dip the smooth curve. IV-B Failure Timing and Segment-Based Annotation Eqs. (3) and (4) need when the rollout went wrong: sfails_fail (or equivalently tfailt_fail) for the smooth head, and intervals (tstart,tend)(t_start,t_end) for the binary head. Rollout logs carry only an episode-level failure flag and a short human comment (e.g., “two patties”, “scratching cheese”, “couldn’t pick lettuce”), neither of which is enough to localize the failure in time, let alone delimit mistake segments. We therefore annotate every flagged rollout by hand. We maintain two annotation files, of increasing fidelity, that together cover the two labeling routes: • A single-point table that records one failure time tfailt_fail per failed episode, alongside a short free-text reason. This is sufficient for Eq. (3) and is what we used in the four-shape VmcV_mc ablation reported in Section VI-D. • A multi-interval table that records the episode outcome and up to three (tstart,tend)(t_start,t_end) mistake segments per episode, again with a free-text reason. This is required for Eq. (4) and lets us annotate both terminally-failed episodes and successful episodes that contained a recovered intermediate mistake. A companion table adds a per-segment severity weight ∈[0,1]∈[0,1] that feeds the multiplicative VmcV_mc dip described in Section IV-A. Annotation takes roughly one minute per episode for the single-point table and two to three minutes for the multi-interval segments; a single sitting (well under two hours) covers the single-point pass over the full 8888-failure set; the multi-interval segments additionally cover the success episodes that contain a worth-flagging recovered mistake (≈70≈\!70 across the six rollout sets, annotated incrementally as each set was reviewed). The free-text reason is parsed by a small keyword matcher that maps phrases such as “couldn’t pick lettuce” → stage 4, “two patties” → stage 1, and “scratching cheese” → stage 2, recovering sfails_fail (counted as the number of fully completed stages, so “couldn’t pick lettuce” means stages 1–4 were done) deterministically for the smooth head; the annotated segment endpoints are used as-is to drive VbinV_bin. We did experiment with using a Gemma-4-26B VLM as an automated stage-completion labeler, prompting it to return the highest completed stage on 1212 probe frames per episode. On our 8888-failure set the best prompt configuration we tried reached only a per-episode mean absolute error of ≈0.21≈\!0.21 on the failure fraction (against the manual annotations as the reference), well above the ≈0.10≈\!0.10 threshold at which we judged it usable as a sole source of truth. The dominant failure mode was the VLM occasionally hallucinating a higher stage than was actually reached, which the per-episode peak heuristic then over-amplified. We retain manual annotation as the only source of sfails_fail and of the mistake intervals used in this paper, and revisit automated labeling as an open problem in Section VIII. IV-C Intervention-Derived Segments The manual annotation of Section IV-B is accurate but does not scale: it is annotated by watching video, and a fleet-scale post-training loop produces far more failures than a person can review. A human-in-the-loop (HIL) teleoperation station already produces the same (tstart,tend)(t_start,t_end) mistake intervals Eq. (4) consumes, for free. When an operator takes over a struggling rollout, the recording carries a per-frame intervention flag; each contiguous run of that flag is a segment in which the operator judged the state bad at tstartt_start and had repaired it by tendt_end. This is a third annotation source, denser and cheaper than either manual file, and it is the one that scales. Two properties of the signal require care, both of which mirror the reaction structure of a takeover. First, the flag is a lagging indicator: the operator perceives the problem and reacts after the causal action, so we shift the VbinV_bin target a fixed lead λ=1.5λ=1.5 s earlier than tstartt_start, and we exclude the ambiguous run-up [tstart−amb,tstart−λ)[t_start-amb,\,t_start-λ) (amb=5=5 s) from the binary loss with a per-frame ignore mask rather than guessing its label. Second, frames inside the takeover are produced by the human, not the policy, so their VmcV_mc regression target is down-weighted (to 0.50.5): they describe the corrective mixture, not the policy being evaluated. The smooth head keeps the same success-then-decay curve of Section IV-A with a multiplicative trapezoid dip over each segment; the sharp head gets a hard mistake window inside the masked interval, exactly as in Eq. (4). Crucially, the intervention labels do not replace the manual annotations; they augment them. We train one model on the union, with the intervention frames capped at ≈20%≈\!20\% of the sampler’s mass and the manually annotated set kept as the sole validation and model-selection signal, so the dense weak labels sharpen the detector without displacing the human-verified target. Episodes whose eventual outcome cannot be confirmed from the session log are held out of training entirely. Section VI-E evaluates what this buys. IV-D Architecture TABLE I: ValueFormer architecture summary for the main dual-head ViT-L/16 setup with (dmodel,L,H,seq_len)=(256,2,4,16)(d_model,L,H,seq\_len)=(256,2,4,16), six-view encoding, and the per-frame binary detection head enabled. Block Definition (shape) # params Frozen encoder 6 views (3 cams + cam_high ROI + 2 wrist ROIs), DINOv3 ViT-L/16 CLS concat (B,S,6144)(B,S,6144) frozen Vision projection Linear(6144→256)(6144→ 256), output (B,S,256)(B,S,256) 1,573,120 State projection Linear(14→256)(14→ 256), output (B,S,256)(B,S,256) 3,840 Time projection Linear(2→256)(2→ 256), output (B,S,256)(B,S,256) 768 Positional embedding Learnable embedding, 16 positions, dim 256 4,096 Causal transformer 2 layers, 4 heads, GELU pre-norm, FFN width 4dmodel4d_model 1,579,520 Final normalization LayerNorm(256)(256) on last-token pathway, output (B,256)(B,256) 512 VmcV_mc head MLP 256→512→256→1256→ 512→ 256→ 1 (smooth value logit) 264,705 VbinV_bin head MLP 256→128→1256→ 128→ 1 (per-frame mistake logit) 33,025 Trainable total ValueFormer (projections + transformer + dual heads) 3,459,586 ValueFormer has three components: the visual frontend, the causal transformer body, and two output heads that share the backbone. Figure 7 traces one prediction end to end through all three. Figure 6: The six views that make up one ValueFormer input frame. Top: the three full camera images (cam_high, cam_left_wrist, cam_right_wrist), each with its region-of-interest (ROI) box overlaid in the matching color. Bottom: the three corresponding ROI crops. The two wrist ROIs cover the contact region where a missed grasp becomes visible several seconds before the top-down view registers the consequence, and the cam_high ROI covers the gripper region of the station. Every view is encoded independently by the same frozen DINOv3 ViT-L/16 backbone and contributes one CLS token, so a frame yields a 6×1024=61446× 1024=6144-dimensional descriptor (Section IV-D). Visual frontend. Each frame is encoded from six views through a single shared, frozen DINOv3 [8] ViT-L/16 backbone at 518×518518× 518 input resolution. The six views are: the three full camera images (cam_high, cam_left_wrist, cam_right_wrist), a fixed ROI crop of the cam_high gripper region, and two wrist-gripper ROI crops (one per wrist camera); Figure 6 shows all six for a single frame. The two wrist ROIs target the contact regions where most mistakes are visible earliest (e.g., a missed grasp shows up in the wrist crop several seconds before the top-down view picks up the consequence). We keep only the CLS token per view, so each frame yields a 66-view concatenated embedding of dimension 6×1024=61446× 1024=6144. DINOv3 weights are loaded through torch.hub with a small conversion routine that fuses the Hugging Face q/k/v projections into the torch.hub fused-qkv format. A ViT-B/16 encoder at 224×224224× 224 resolution is also supported, as is a legacy four-view setup (3 cameras + cam_high ROI only) under which the labeling-scheme ablation of Section VI-D was run. All other reported runs use the six-view ViT-L/16 configuration. Freezing DINOv3 throughout has two benefits: the dataset-build step becomes embarrassingly parallel and caches to disk, and its self-supervised features transfer robustly across embodiments with no task-specific fine-tuning. Figure 7: ValueFormer data flow for one prediction. The six views of Figure 6 pass through the shared frozen DINOv3 ViT-L/16 encoder (images only) and their CLS tokens are concatenated to 61446144 dimensions per frame; the 1616-frame (88 s) window is left-padded with the first frame when fewer than 1616 samples are available. The joint state (1414-dimensional, 77 per arm) and the two time features bypass the encoder entirely and are projected separately, after which all three modalities are combined by element-wise sum rather than concatenation, so vision cannot dominate through its much larger raw dimensionality. A learnable positional embedding, the causal transformer body (position t attends only to positions 0…t0… t), and a final LayerNorm feed the output heads. All shapes agree with Table I. The diagram is drawn for the single-head recipe, where the second head is an optional episode-level success / failure classifier; the dual-head configuration used for the reported results keeps that same 256→128→1256→ 128→ 1 head but supervises it per frame as VbinV_bin. Figure 8: Why the transformer body is causal. Attention for a four-frame window under a bidirectional encoder (left) versus the causal, decoder-only mask ValueFormer uses (right). Top: the mask grid, where each row is a query frame, each column a key frame, and a filled cell marks an allowed attention edge. Bottom: the same rule drawn as arrows from a single query frame t3t_3. The bidirectional encoder lets every frame attend to all others, the future included, whereas the causal mask restricts frame t to positions ≤t≤ t. Only the causal pattern is admissible for the online per-frame value prediction of this section, since future frames are unavailable at inference time. Transformer body. Let t∈ℝ6144z_t ^6144 denote the frame embedding, t∈ℝ14q_t ^14 the joint state, and ϕt=(ϕtsec,ϕtfrac)∈ℝ2 φ_t=(φ^sec_t,φ^frac_t) ^2 the time features (raw seconds and normalized fraction into the episode). Three linear projections map each modality to dmodeld_model: t=Wvist+Wstatet+Wtimeϕt.x_t=W_vis\,z_t+W_state\,q_t+W_time\, φ_t~. (5) Element-wise addition rather than concatenation equalizes the contribution of each modality; in our early experiments, concatenation let vision dominate because of its much larger raw dimensionality. A learnable 1616-position embedding is added, and a causal TransformerEncoder (L=2L=2 layers, H=4H=4 heads, GELU, pre-norm, feed-forward dimension 4dmodel4d_model, dropout 0.20.2 in the body and 0.40.4 on the CLS path that feeds the binary head) runs over an 88 s context window (1616 samples at 22 Hz). The body is causal: a triangular attention mask (Figure 8) lets frame t attend only to frames at or before t, so the value emitted at time t is a strict function of past and present frames and never of the future. This is the one structural difference from a bidirectional encoder, and it is what makes the model valid for online, per-frame prediction at policy rate, where future frames are simply not yet available. The last-token output is LayerNorm’d before being passed to the two heads. We shrink the context from the 3232-sample (1616 s) window used in early experiments because online detection benefits from sharper transitions: a longer window smooths the value over the half-window scale, which is desirable for VmcV_mc but actively masks the moment a mistake begins for VbinV_bin. The window length is the most impactful hyperparameter we tuned, and we swept over seq_len∈8,16,32,64seq\_len∈\8,16,32,64\. Heads. The smooth value head VmcV_mc is a 3-layer MLP (256→512→256→1256→ 512→ 256→ 1) with LayerNorm, GELU, and dropout, followed by a sigmoid. The binary detection head VbinV_bin is a separate 2-layer MLP (256→128→1256→ 128→ 1) that branches off the same shared last-token representation and is also sigmoid-activated; it is supervised by the per-frame target of Eq. (4). The two heads remain active at inference time, so a single forward pass returns both a smooth critic value and a sharp detection probability per frame. In an earlier single-head configuration the auxiliary head was trained on an episode-level success / failure flag and used only as a regularizer; the dual head we describe here turns that same MLP shape into a deployable per-frame detector by replacing the target. IV-E Training Objective We train both heads jointly with binary cross-entropy. Given per-sample MC label vi∈[0,1]v_i∈[0,1], per-frame binary label vibin∈0,1v^bin_i∈\0,1\, predicted MC logit ℓ^v _v, and predicted binary logit ℓ^b _b, ℒmc=−1B∑iwi(vilogσ(ℓ^v,i)+(1−vi)log(1−σ(ℓ^v,i))),L_mc=- 1B _iw_i (v_i σ( _v,i)+(1-v_i) (1-σ( _v,i) ) )~, (6) ℒbin=BCEw(ℓ^b,vbin;α+),ℒ=ℒmc+βℒbin,L_bin=BCE_w ( _b,v^bin;\, _+ )~, =L_mc+β\,L_bin~, (7) where wi=1+(wfail−1)vi=0w_i=1+(w_fail-1)1\v_i=0\ optionally up-weights flat-zero frames, α+=clip(n0/n1, 0.1, 10) _+=clip(n_0/n_1,\,0.1,\,10), where n0n_0 and n1n_1 are the per-batch counts of mistake (vbin=0v^bin=0) and good (vbin=1v^bin=1) frames, is the batch-wise pos-weight that balances the ∼95%/5% \!95\%/5\% good-state/mistake-state imbalance per batch (clipped to avoid pathological mini-batches), and β=1.0β=1.0 balances the binary term. The unit weight is a deliberate change from our earlier single-head recipe, in which the (then episode-level) auxiliary loss was a tiny regularizer at β=0.1β=0.1; the new VbinV_bin target is now a primary output rather than a regularizer and is weighted accordingly. BCE-on-value is preferable to MSE-after-sigmoid here because the composed sigmoid-MSE loss saturates near the 0,1\0,1\ endpoints, which are precisely the regime where success and failure diverge most. We train exclusively with the BCE objective in this paper and report final validation MSE only as a measurement, not a training signal. Optimization uses AdamW with lr=10−4lr=10^-4, weight decay 0.050.05, five epochs of linear warm-up, and cosine annealing to zero over the remaining epochs. The batch size is 256256. A 4-group balanced sampler over expert-success,expert-fail,rollout-success,rollout-fail\expert-success,expert-fail,rollout-success,rollout-fail\ equalizes the weight of each data-source group regardless of its absolute count. In our dataset the expert-fail group is empty (expert teleoperators do not fail often enough to record a meaningful sample), so the sampler reduces to three groups that up-weight the ∼88 \!88 failure rollouts against the ∼1,249 \!1,249 expert successes. We train for at most 100100 epochs with early stopping on validation MSE, patience 1515. V Experimental Setup V-A Task and Embodiment All experiments are on a real-robot bimanual sandwich-assembly task. Two Trossen WidowX arms with seven degrees of freedom each are mounted on a dedicated station with six ingredient bins. A π0 _0-family VLA policy drives the arms end-to-end in response to the natural-language prompt “make a sandwich”. The nominal task has six ordered stages: bottom_bun, patty, cheese, tomato, lettuce, top_bun. A rollout is considered successful if all six ingredients are placed correctly and the sandwich is closed. In practice, the failure modes we observe most often are retry (the policy fails a grasp and recovers), double-placement (two patties stacked), wrong-stage (scratching the cheese bin without grasping), and stall (the policy freezes mid-stage). V-B Data Figure 9: Dataset composition: expert successes, rollout successes, and rollout failures. Full details in Section V-B. Two datasets recorded on the same physical station in LeRobot v2 format [7] make up the training corpus: a teleoperated expert set of 1,2491,249 successful episodes (collected by human operators at the station), and a policy rollout set of 178178 autonomous rollouts split 9090 success / 8888 failure. A smaller third set of held-out episodes is reserved for the qualitative plots in Section VI-C and is never seen at training time. Each episode runs for ∼2,200 \!2,200 frames at 3030 fps; we subsample every 1515 frames (0.50.5 s stride), giving ∼150 \!150 samples per episode and 213,102213,102 samples in total. The train / val split is by-episode (no frame leakage), at 181,421181,421 / 31,68131,681 samples. Figure 9 summarizes the composition. Feature extraction caches a six-view embedding per sample to disk: each frame is independently passed through the frozen DINOv3 ViT-L/16 backbone for the three full cameras and the three gripper ROIs of Section IV-D, and the per-view CLS tokens are concatenated to a 61446144-dim vector. The total feature payload is ∼6.5 \!6.5 GB; rebuilding from raw videos takes ∼90 \!90 min on an RTX-5090-class GPU and dominates the end-to-end cost (training itself runs in under three minutes). An earlier four-view feature build (3 cameras + cam_high ROI only, 40964096-dim) is also retained and is the substrate for the labeling-scheme ablation reported in Section VI-D; we describe that ablation under the four-view single-head recipe so that the labeling effect we isolate is not confounded with the backbone-width change. Annotation files (Section IV-B) live alongside the LeRobot dataset and feed two parallel labeling routes. The legacy single-point table records one tfailt_fail per failed rollout (8888 episodes); under this route VbinV_bin is reduced to t<tfail1\t<t_fail\ per frame, and this is what the headline dual-head configuration of Table I is trained on. The multi-interval table is a strict superset that additionally annotates segment intervals on every flagged rollout, including 7070 otherwise-successful rollouts that contain a recovered intermediate mistake (typically a 33–88 s retry on the lettuce or cheese stage) and a handful of failed rollouts with two or three distinct mistake events, for 196196 mistake intervals in total. The interval representation contributes per-frame Vbin=0V_bin=0 supervision that the single-point table cannot, because a recovered mistake on an episode that ultimately succeeds has no tfailt_fail to point at; we use the segment-based labels in a binary-only ablation configuration and rely on the single-point table for the dual-head numbers reported here. V-C Labeling-Scheme Comparison To demonstrate the effectiveness of the proposed labeling approach, we conduct a controlled comparison in which the model architecture, training dataset, and training recipe are all held fixed, and the only variable is the per-frame value label assigned to frames in failed episodes. Under this setup, any difference in performance can be attributed solely to the choice of fail-episode labeling scheme, isolating its effect from confounding factors such as model capacity, data composition, or optimization. We compare five concrete fail-episode label shapes; success-episode labels follow Eq. (2) in every case. Let vksucc=γN−1−kv^succ_k=γ^N-1-k, c=sfail/Nsc=s_fail/N_s be the partial-credit factor, and kfail=⌊c⋅N⌋k_fail= c· N . The five shapes are: • MC-smooth (ours; Eq. (3)): vkfail=vksuccv^fail_k=v^succ_k for k≤kfailk≤ k_fail and vkfail=vkfailsucc⋅γ(k−kfail)v^fail_k=v^succ_k_fail·γ^(k-k_fail) for k>kfailk>k_fail. Pre-failure follows the success curve; post-failure decays smoothly. • A: outcome-scaled: vkfail=c⋅vksuccv^fail_k=c· v^succ_k for k<kfailk<k_fail and 0 otherwise. The whole pre-failure curve is scaled by the partial-credit factor, so v0fail=c⋅γN−1v^fail_0=c·γ^N-1 already encodes the eventual outcome at t=0t=0. • B: cliff: vkfail=vksuccv^fail_k=v^succ_k for k<kfailk<k_fail and 0 otherwise. Pre-failure is identical to success, but the label drops to 0 in a single step. • C-linear (α-mix): with α=k/kfailα=k/k_fail, vkfail=(1−α)⋅vksucc+α⋅cv^fail_k=(1-α)· v^succ_k+α· c for k<kfailk<k_fail and 0 otherwise. The label starts at v0succv^succ_0 and linearly decays to c by kfailk_fail. • C-late: pre-failure follows vksuccv^succ_k until kstart=⌊(1−ρ)⋅kfail⌋k_start= (1-ρ)· k_fail with ρ=0.25ρ=0.25, then linearly ramps to c at kfailk_fail, and is 0 afterwards. A graphical comparison of the four ablation shapes (A, B, C-linear, C-late) for three representative failure stages is shown in Figure 5; the MC-smooth shape itself is shown in the right panel of Figure 4. Apart from the failure label, every other detail (DINOv3 ViT-L/16 backbone in the four-view configuration of Section V-B, 3232-frame causal window, two-layer transformer body, BCE objective with the episode-level auxiliary classifier of our earlier single-head recipe, AdamW with warm-up + cosine schedule, balanced sampler, 100100-epoch budget with patience-1515 early stopping) is held constant. Each scheme is trained as a separate run from the same initialization, and the model checkpoint with the best validation MSE on the original target is used for evaluation. We run the labeling ablation under the legacy single-head recipe rather than the six-view dual-head configuration of Table I so that the effect we isolate is purely the VmcV_mc label shape, uncontaminated by the architectural changes (extra views, per-frame VbinV_bin supervision, shorter 88 s window) that the dual head also brings. The arguments that motivate the MC-smooth choice (no outcome leakage at t=0t=0 and no discontinuity at kfailk_fail) are properties of the label itself and are independent of the encoder width or window length, so we expect the ordering to carry over; re-running the full five-scheme ablation under the dual-head recipe is the cleanest piece of follow-up work and is left for a future iteration. V-D Metrics We report five metrics on the held-out set: validation MSE (direct regression loss), MAE, mean-V-on-success and mean-V-on-fail (sanity checks that the network has actually separated the two classes), and the separation ΔV=V¯succ−V¯fail V= V_succ- V_fail. For the auxiliary head we examine its qualitative effect on the failure-class predictions (Section VI-D). For qualitative analysis we plot per-episode predicted-versus-MC-label curves. VI Results VI-A Quantitative Table I gives the headline numbers for the main configuration: MC-smooth stage-aware labels, DINOv3 ViT-L/16 backbone, and the per-frame binary detection head (VbinV_bin) enabled. At the best checkpoint the model reaches validation MSE 3.0×10−43.0× 10^-4, MAE 0.0150.015, mean-V-success 0.5370.537, mean-V-fail 0.0000.000, and separation ΔV=0.537 V=0.537. In practice, the most visible effect before we reached this configuration was the gap between MC-smooth and any of the four alternative fail-episode shapes ablated in Section VI-D: under the naive flat-zero baseline (a special case of A with c=0c=0) the predictions collapse to zero on clean successes and the training curves do not stabilize. TABLE I: Held-out value-prediction metrics for the main ValueFormer configuration. Setup and metrics are defined in Section V. Quantity Value Validation MSE 3.0×10−43.0×10^-4 Validation MAE 0.0150.015 Mean-V-success 0.5370.537 Mean-V-fail 0.0000.000 Separation ΔV V 0.5370.537 Epochs to best checkpoint ∼80 \!80 Wall-clock training time <3<\!3 min VI-B Training Dynamics Across the 100100-epoch training run, validation MSE drops from ≈1.4×10−3≈\!1.4× 10^-3 at initialization to ≈3×10−4≈\!3× 10^-4 within the first 8080 epochs; MAE follows the same shape, from 0.0260.026 down to 0.0150.015. Mean-V-success and separation both stabilize around 0.540.54; mean-V-fail sits at 0.0000.000 once the stage-aware labels take over. The binary detection-head loss ℒbinL_bin converges within the first handful of epochs and does not dominate training. The entire run completes in under three minutes on a single RTX-5090-class GPU; the runtime is dominated by the dataset-loading overhead rather than by the transformer itself. VI-C Qualitative: The Four Rollout Signatures Figure 10: Four canonical rollout types on held-out episodes: clean success, success-with-retry, early collapse, and stuck-scratching. See Section VI-C. Figure 10 shows the four rollout signatures on unseen episodes; ValueFormer reproduces each of them despite never having seen these specific episodes at training time. On a clean success the predicted V(s)V(s) is a smooth rising curve that tracks the MC label almost exactly. On a success-with-retry episode the robot has one or more failed grasp attempts at a specific stage (lettuce, in the example); V(s)V(s) dips during the retry and recovers as soon as the stage completes. This is the behavior we most wanted, because it gives a supervisor early warning before a retry turns into a full collapse. On an early-collapse rollout (two cheese slices placed instead of one), V(s)V(s) rises through the bottom-bun and patty stages, peaks near 0.40.4, and then decays as the second cheese is placed. On a stuck-scratching rollout the policy never commits a successful grasp, and V(s)V(s) stays low throughout. Figure 11 overlays the ValueFormer prediction and the MC label on six representative held-out episodes. Agreement is tight on successes; on failures, the predicted V(s)V(s) lags the MC-label transition by several seconds. We attribute this qualitatively to the causal history window (these overlays use the earlier single-head 3232-frame/1616 s configuration; the dual-head main setup halves it to 1616 frames/88 s): the model first sees the failure stage only after a fraction of its history has observed it. We did not measure the lag precisely; visually, across the episodes in Figure 11, it looks to be on the order of 55 to 1010 s. Figure 11: Per-episode predicted V(s)V(s) (blue) vs. MC label (green dashed) curves on held-out episodes, with the clean-success reference for the same episode length (gray dotted) shown for the two failures. Green title: success; red title: failure. The dual-head model emits VmcV_mc and VbinV_bin in a single forward pass, and the two curves trade off as designed. The smooth VmcV_mc output is the curve shown in Figures 10 and 11 and tracks gradual progress; VbinV_bin is binary-like and snaps near zero only over annotated mistake segments, which makes it a more actionable abort signal for an operator but visually less informative on the kinds of recovered-mistake episodes that Figure 10 highlights. Throughout, our monitor visualizations (Figures 1 and 15) draw VmcV_mc in green and the failure probability 1−Vbin1-V_bin in orange on the same time axis, so an operator can read both at once. VI-D Labeling-Scheme Ablation The labeling scheme is the single most visible design choice in the pipeline. We trained one ValueFormer per scheme of Section V (MC-smooth, A, B, C-linear, C-late) under the identical recipe and evaluated each on the held-out rollout set (1212 success, 99 failure rollouts). Table I reports the original-target validation loss, rollout MAE, the mean predicted V on success and on failure rollouts, and the success/failure separation ΔV=V¯succ−V¯fail V= V_succ- V_fail. TABLE I: Held-out rollout metrics for the five label shapes of Section V, all trained in the legacy four-view single-head configuration of Section V-B (ViT-L/16 backbone) so that only the label shape differs between the five runs. Validation BCE loss is on the scheme’s own target; all other columns are computed on the same 1212-success / 99-failure rollout set. Lower is better for loss / MAE; higher is better for ΔV V. Note that the rollout MAE here is not comparable to the validation MAE of Table I, which is measured on the validation split under the main six-view dual-head configuration. Scheme Val loss Rollout MAE V¯succ V_succ V¯fail V_fail ΔV V MC-smooth (ours) 0.00051 0.024 0.533 0.390 0.143 A: outcome-scaled 0.00192 0.058 0.504 0.195 0.309 B: cliff 0.00202 0.047 0.529 0.275 0.254 C-linear 0.00190 0.058 0.515 0.266 0.249 C-late 0.00178 0.048 0.516 0.267 0.250 Figure 12: Per-episode predicted V (blue) versus the scheme’s own MC label (green dashed) on six held-out rollouts (columns) under the five label shapes (rows: MC-smooth, A: outcome-scaled, B: cliff, C-linear, C-late). MC-smooth tracks the success target tightly on the success columns and decays smoothly on the failure columns; A and C-linear visibly distort the pre-failure prefix, B exhibits the cliff drop with high-frequency oscillation just before kfailk_fail, and C-late mostly tracks the success curve but pays a coarser price on long failure tails. When evaluated as a regression problem, MC-smooth wins by roughly a factor of three on validation BCE loss and a factor of two on rollout MAE: its pre-failure prefix is identical to the success curve, so the network is never asked to regress two visually identical prefixes onto two different targets, which is the same pathology Section I-C identifies for the flat-zero baseline. When evaluated as a binary detector (the ΔV V column), the picture is less flattering for MC-smooth: outcome-scaled (A) achieves the largest separation, 0.3090.309, with B/C-linear/C-late clustered around 0.250.25 and MC-smooth a smaller 0.1430.143. We argue this is a label artifact rather than a real advantage. In A, the entire pre-failure curve is multiplied by c=sfail/Nsc=s_fail/N_s, which is determined only by what happens at the end of the episode; consequently v0fail=c⋅γN−1v^fail_0=c·γ^N-1 already differs from v0succv^succ_0 at the very first frame, before any visible failure cue. This is a form of label leakage: the t=0t=0 label depends on the eventual outcome of the episode, but at deployment the value head sees only an ongoing rollout, with no access to whether it will end in success or failure, so a target conditioned on the final outcome is not learnable from the observation alone. The network is in effect rewarded for guessing the eventual outcome from frame 0 alone, a target that cannot be derived from the observation. The same logic applies to a milder degree to C-linear, whose label starts at v0succv^succ_0 but begins decaying immediately. The cliff scheme (B) has no t=0t=0 leakage but introduces a discontinuity at kfailk_fail that the value head is forced to fit with a near-step function; visually this surfaces as a high-frequency oscillation just before the cliff in Figure 12. C-late comes closest to MC-smooth in spirit, only diverging in the last 25%25\% of the pre-failure window, but pays a coarser price on long failure tails because the label is hard-zero past kfailk_fail rather than smoothly decayed. A closer look at Figure 12 shows that the kind of separation A and C-late achieve is not the kind we want. On the failure rollouts (the bottom rows of the grid), both schemes drive the predicted V to near zero on every frame past kfailk_fail, regardless of how much of the task had been completed by that point. This is a direct consequence of the label: A and C-late both set vkfail=0v^fail_k=0 for k>kfailk>k_fail, so on the post-failure tail the value head is in effect trained as a binary classifier (failure ⇒0 0), and an episode that collapses after four correct ingredients is rendered indistinguishable from one that fails at the bottom-bun stage. MC-smooth, by contrast, decays from vkfailsuccv^succ_k_fail rather than from zero, so the post-failure value automatically carries how far the policy got: a late failure leaves a higher residual tail than an early one, without the network having to be told the outcome at t=0t=0. The smaller headline ΔV V for MC-smooth is therefore a feature of the label rather than a deficiency of the model. A’s inflated ΔV V is the joint effect of (i) leaking the outcome into the pre-failure curve via the c-scaling and (i) hard-zeroing the post-failure tail; either failure mode alone is undesirable, and the combination is what produces the largest but least informative separation in Table I. In sum, MC-smooth is the only scheme that simultaneously (i) has v0fail=v0succv^fail_0=v^succ_0, ruling out outcome-leakage at t=0t=0, (i) has no discontinuity at kfailk_fail, ruling out the high-frequency artifact visible under B, and (i) assigns smoothly decaying credit after the failure rather than a hard zero, matching the intuition that a robot whose mistakes accumulate gradually should see its value drop gradually rather than collapse. The fact that it is also the strongest by validation BCE loss and rollout MAE is what motivates our choice. The narrower ΔV V on the held-out rollout set is a design trade-off rather than a regression: a downstream binary supervisor can still threshold V(s)V(s), and the smooth shape preserves the rich within-episode dynamics (dip-and-recover on retries, peak-and-drop on early collapse) that a pure detector cannot. We also observe that the auxiliary classifier head, while it contributes only a small change in validation MSE, has a visibly consistent effect on failure-class predictions: with the auxiliary head the predicted values on failure rollouts concentrate near zero with tighter dips, as seen in the failure panels of Figure 11. Without it, we observed a small positive residual in the failure-class mean during early experiments, though we do not report a calibrated number here. VI-E Supervision from Human Interventions We test the intervention-derived labels of Section IV-C on a separate HIL dataset: 134134 policy rollouts of the same task (362362k frames) carrying the per-frame intervention flag, with 333333 takeover segments across 129129 episodes. We hold out 2424 of those episodes for validation (never trained under any variant), leaving 105105 for training, and compare two critics under an otherwise identical recipe: one trained on the manual annotations alone, and one trained jointly on the manual annotations plus the intervention labels. Because the two models saturate to different degrees, we compare them at a matched 5%5\% per-frame false-positive rate, and we also report the two threshold-free ranking metrics (AUROC, average precision) that do not depend on an operating point. Detection “lead” is the time between the first sustained alert and the true takeover onset, a label-independent quantity, since the onset comes from the raw flag, not from our shifted target. TABLE IV: Held-out failure detection on the HIL set (2424 episodes), manual annotations only vs. manual ++ intervention supervision, at a matched 5%5\% per-frame false-positive rate. The deployed model is a 22-seed ensemble of the best configuration (seq_len=16seq\_len=16, intervention share 0.20.2); across a 5454-run sweep, every manual++intervention configuration outscored every manual-only one on held-out AP. Manual Manual ++ intervention Metric only single ensemble Average precision 0.380.38 0.790.79 0.820.82 AUROC 0.690.69 0.930.93 0.940.94 Detected before takeover 35%35\% 95%95\% %95\% Median lead 1.21.2 s 2.42.4 s 2.42.4 s Manual-val MSE 10−310^-3 10−310^-3 10−310^-3 Table IV shows the effect: average precision rises 0.38→0.820.38→ 0.82 and the fraction of held-out mistakes flagged before the operator reacts goes from 35%35\% to 95%95\%, at the same false-alarm budget and with no change in the manually validated smooth-head accuracy. Figure 13 confirms the gain is threshold-free: the manual++intervention curve dominates the manual-only one across the whole precision–recall and ROC range. The improvement is not a lucky hyperparameter: over a sweep of label shape, intervention share, context length, and detection-head weight, all 4848 manual++intervention runs beat all 66 manual-only runs on held-out AP, with a clean gap between the two clusters (Figure 14). Figure 13: Threshold-free detection on the 2424 held-out HIL episodes. The manual++intervention critic (blue) dominates the manual-only critic (gray) across the whole precision–recall (left) and ROC (right) range: AP 0.38→0.820.38→ 0.82, AUROC 0.69→0.940.69→ 0.94. Figure 14: Every manual++intervention configuration (4848 runs, blue) beats every manual-only one (66 runs, gray) on held-out average precision, with a clean gap. The effect is robust to label shape, intervention share, context length, and detection-head weight. Qualitatively, the dual head behaves as designed on unseen episodes. Figure 15 traces four held-out rollouts spanning one to four takeovers: the binary head’s failure probability rises into each takeover and falls back once the state is repaired, while the smooth value dips and recovers with each mistake and climbs toward 11 as the sandwich completes. This is the recovered-mistake, dip-and-recovery structure that the terminal single-point labels of Section IV-B cannot express, now supplied at fleet scale by the intervention flag. The one caveat is that this held-out ground truth is itself intervention-derived (with the λ=1.5λ=1.5 s reaction-lag correction), so it measures agreement with where the operator intervened rather than an independent notion of failure; the lead-time column, measured against the raw onset, is the exception. A fully independent evaluation (hand-labeled mistake onsets on the held-out episodes) is the natural next measurement. Figure 15: The dual head on four held-out HIL episodes (11–44 takeovers). Predicted value VmcV_mc (green) and failure probability 1−Vbin1-V_bin (orange); each shaded band is a human takeover and the red line marks its onset. The failure probability spikes into every takeover, while the value dips as the rollout drifts and recovers once the operator has repaired it. That dip-and-recovery is exactly the good → bad → good gradient a terminal pass / fail label cannot express. VI-F From Critic to Policy-Training Weights The critic’s outputs admit three distinct per-frame policy-training weights, set against the flag-only IWR control (intervention-weighted regression: up-weight the human-takeover frames in the flow-matching loss, with no critic in the loop); keeping them apart matters because they draw on different heads. All four schemes (the three critic-derived weights plus the IWR control) floor human-takeover frames at w=9w=9 and leave takeover-free episodes untouched; they differ only on the autonomous frames of takeover episodes (Table V, Figure 16). (i) IWR: the intervention flag alone, w∈1,9w∈\1,9\, with no critic. (i) vf-mask: discrete, score-based, with w=0w=0 where Vbin<0.5V_bin<0.5 flags a failing autonomous frame, the learned replacement for a fixed pre-takeover masking window. (i) soft: continuous, score-based, with w=0.1+0.9Vbinw=0.1+0.9\,V_bin, a smooth version of the mask with no threshold cliff. (iv) awr: continuous, advantage-based, using w=clip(eAt/βawr,0.1,9)w=clip(e^A_t/ _awr,0.1,9) with At=Vmc(t+HA)−Vmc(t)A_t=V_mc(t+H_A)-V_mc(t) (lookahead HA=50H_A=50 frames, temperature βawr=0.3 _awr=0.3); near 11 on steady progress by construction and responsive only at value swings. Two of the three are now trained into the policy and evaluated end-to-end on the physical station (vf-mask and scheme (iv), hereafter vf-awr), and we report the controlled A/B in Section VI-G. Scheme (i), soft, is implemented but not yet trained. For the vf-awr dataset the autonomous weights are continuous with median 1.031.03 (range 0.130.13–7.617.61), confirming the near-11 design intent, and its intervention gradient share peff=0.459p_eff=0.459 is matched to the IWR control’s (≈0.46≈\!0.46), so the A/B isolates the advantage redistribution rather than a rebalancing effect. Scheme (iv) is worth isolating because it is the one rung on this list that is genuinely a reinforcement-learning update rather than reweighted imitation: advantage-weighted regression is the closed-form solution of the KL-constrained policy-improvement problem maxπ[A(s,a)] _πE[A(s,a)] s.t. KL(π∥πBC)≤ϵKL(π\,\|\, _BC)≤ε, so weighting the flow-matching loss by eAt/βawre^A_t/ _awr performs one step of offline policy iteration with VmcV_mc as the policy-evaluation critic. Under this lens IWR is the degenerate special case (a constant log-w “advantage” on human frames, zero elsewhere), and advantage conditioning rather than weighting is the natural next rung. TABLE V: Per-frame policy-training weight on autonomous frames of takeover episodes. All schemes set w=9w=9 on takeover frames. Scheme Signal Weight w Status IWR flag only 11 evaluated (control) vf-mask VbinV_bin [Vbin≥0.5]1[V_bin≥ 0.5] evaluated (Section VI-G) soft VbinV_bin 0.1+0.9Vbin0.1+0.9\,V_bin implemented only awr AtA_t clip(eAt/βawr,0.1,9)clip(e^A_t/ _awr,0.1,9) evaluated (Section VI-G) Figure 16: How the critic’s outputs become per-frame training weights, on one training episode; shaded spans are human takeovers. Top row: the two value-network outputs that drive everything below, VmcV_mc (violet) and the failure probability 1−Vbin1-V_bin (orange). Lower three rows: the three critic-derived weights on a logarithmic weight axis, each plotted against IWR’s discrete 1/91/9 step (dotted). vf-mask zeroes flagged-failing autonomous frames (drawn at the axis floor); soft ramps smoothly toward 0.10.1 on them; awr hovers near 11 (advantage ≈0≈ 0 under steady progress) and moves only at value swings. All three floor human-takeover frames at w=9w=9. Before running the A/Bs we recorded a prediction, which Section VI-G then tests. vf-mask was the most likely gain (it repairs the fixed window’s documented timing bluntness at matched coverage); soft the robust runner-up (mis-calibration down-weights rather than deletes); and vf-awr the least likely to move results on its own (its weight barely departs from 11 on most frames and the horizon-differenced advantage is the critic’s noisiest output), though it supplies the same AtA_t that drives advantage conditioning, the natural next step beyond reweighting. VI-G Hardware A/B: Critic-Derived Weights vs. Flag-Only IWR We now test whether a critic-derived per-frame weight beats the flag-only intervention baseline when everything else is held fixed. Both critic variants train on the same 985985k-frame merged set as the flag-only IWR control (the 134134-episode HIL corrections of Section VI-E, with no-motion operator-pause frames excised, merged 2:12:1 with replayed base autonomous data) and keep the same w=9w=9 floor on human-takeover frames; the only thing that changes is the weight on the autonomous frames of takeover episodes. The control leaves them at 11, vf-mask zeroes the 9.6%9.6\% that VbinV_bin flags as failing, and vf-awr scales them by the clipped advantage (median 1.031.03). Each A/B therefore isolates the critic-derived weight against a same-data control. The deployed policy is the underlying π0.5 _0.5 VLA; ValueFormer supplies only the per-frame training weight and is not itself part of the policy at inference. Each variant is scored over 1010 trials of two consecutive sandwiches (2020 sandwiches), with the six subtasks each graded in [0,1][0,1] and a binary completion flag per sandwich (all six ingredients placed, one per stage). We refer to one graded subtask of one sandwich as a cell, so each variant is scored over 20×6=12020× 6=120 cells. Table VI reports overall completion with Wilson 95%95\% intervals, the second-sandwich completion rate (the harder, second-in-a-row sandwich), the pooled per-subtask mean, the fraction of cells scored perfectly (clean cells, score 1.01.0: the ingredient was placed correctly with no retry or damage), the count of catastrophic cells (score ≈0≈\!0: the ingredient was missed entirely or wrecked, for example a skipped stage or a crushed patty, so that cell contributes essentially nothing), the count of repeat-picks (the recovery reflex re-running an already-successful pick, a behavioral failure mode inherited from the takeover data), and the median sandwich duration. Clean and catastrophic cells are therefore the two tails of the same per-cell score distribution, and the pooled per-subtask mean is its average. Why two sandwiches in a row. Every trial assembles two consecutive sandwiches with no scene reset in between, and that protocol is deliberate. A single sandwich from a freshly staged station is easier and already reliably achievable for the policy, so it discriminates poorly between variants. The consecutive setting is the harder test because performance is highly sensitive to scene staging, which is difficult to quantify and to reproduce when the ingredients are deformable and the bins evolve over a session. The policy’s own actions compound that drift: a patty tipped while the first one was picked, or a cheese stack pushed out of position, leaves the second sandwich measurably harder. The second-sandwich rate is therefore a natural form of hard-example mining and the most sensitive column in Table VI. TABLE VI: On-robot A/B of the two critic-derived weights against the flag-only IWR control, on identical data and with the same w=9w=9 takeover floor (2020 sandwiches each). Higher is better except for catastrophic cells, repeat-picks, and duration. Best in bold. Metric IWR (control) vf-mask vf-awr Completion 70%70\% (14/2014/20) %85\% (17/2017/20) %85\% (17/2017/20) Wilson 95%95\% [48,85][48,85] [64,95][64,95] [64,95][64,95] Sandwich 22 only 60%60\% %90\% 80%80\% Mean subtask 0.910.91 0.940.94 0.960.96 Clean cells 88%88\% 88%88\% %92\% Catastrophic cells 77 2 2 Repeat-picks 44 0 0 Median duration :1:15 1:211:21 1:221:22 Figure 17: The critic-derived weights against the flag-only IWR control. (Left) per-subtask mean score, pooled over both sandwiches, with a MEAN column; boxed cells mark the tomato/lettuce stages the control is soft on. vf-mask lifts them to 0.97/0.950.97/0.95 and vf-awr to a perfect 1.00/1.001.00/1.00. (Right) overall completion (Wilson 95%95\% CI) and the sandwich-1 / sandwich-2 split: both critic weights reach 85%85\%; vf-mask improves on the harder second sandwich (60%→90%60\%→ 90\%) while vf-awr dips (90%→80%90\%→ 80\%). See Section VI-G. Both critic weights reach the same headline: completion rises from 14/2014/20 (70%70\%) to 17/2017/20 (85%85\%), catastrophic cells fall 7→27→ 2, and repeat-picks fall 4→04→ 0 (Table VI). The two variants get there differently. vf-mask buys the flattest floor: no subtask below 0.890.89, lifting the three the control was soft on (tomato 0.82→0.970.82→ 0.97, lettuce 0.82→0.950.82→ 0.95, top bun 0.88→0.920.88→ 0.92) while giving back a little on cheese (0.93→0.890.93→ 0.89) and bottom bun (1.00→0.941.00→ 0.94), and it is the only variant that improves on the harder second sandwich (60%→90%60\%→ 90\%). vf-awr buys the highest ceiling: perfect tomato and lettuce (1.00/1.001.00/1.00) and the highest mean subtask score of any variant (0.960.96), at the cost of a softer cheese (0.840.84); it is also the cleanest by raw execution (92%92\% perfect cells) and fails gently when it fails (mean 0.860.86 even on its failed sandwiches, vs. vf-mask’s 0.780.78), but unlike vf-mask its second-sandwich rate dips (90%→80%90\%→ 80\%) because it starts higher on sandwich 11. Figure 17 shows both the per-subtask trade-off and the completion split. That two different critic-derived weights, a hard mask and a soft advantage, land the same 85%85\%/zero-repeat-pick result is stronger corroboration than either alone that the lever is the critic, not a lucky threshold. The repeat-pick elimination is the clearest signal: the behavioral reflex survived every flag-only lever (more correction data, a doubled replay share, pause-frame removal: it still fired 44 times in the control) yet does not appear once in 2020 sandwiches under either critic weight. A plausible mechanism, consistent with the traces of Figure 15: the run-up to a redundant re-pick is exactly the low-value drift the critic flags, so vf-mask zeroes it and vf-awr down-weights it; those frames leave the gradient and the reflex is no longer rehearsed, while the takeover frames that teach the recovery motion keep their full w=9w=9. The predictions recorded in Section VI-F came out split. The vf-mask call held cleanly. The vf-awr call did not: we expected the near-11 advantage weight to barely move results on its own, and it instead matched vf-mask’s completion and posted the highest per-subtask quality of any variant. We report this as a prediction that came out wrong in vf-awr’s favor rather than quietly revise it. Statistics and caveats. These results are suggestive, not established. At n=20n=20 the completion differences sit within noise for both variants (Fisher exact p=0.45p=0.45 vs. the control), and several headline gains partly restate the same handful of sandwiches: the catastrophic-cell drop largely tracks the completion count, and on completed sandwiches the per-subtask means converge (0.970.97 for vf-mask, 0.980.98 for vf-awr, vs. 0.990.99 for the control). What the design has going for it is the isolation (same data, same w=9w=9 floor, only the autonomous-frame weight changed) and two quasi-independent channels that agree: the repeat-pick count and the fact that two different critic weights reach the same result. Both variants were evaluated one to two weeks after the control, so ingredient staging (e.g., sticky cheese) is uncontrolled across sessions, and the note-derived counts are heuristic. A paired re-evaluation (shared initial conditions, McNemar test) against the control is the power fix, and the soft variant of Table V is the last member of the family still to run. VI-H Online Serving Cost and Inference-Time Optimization Running the critic live during a rollout, as the operator-facing monitor of Section VI-E, adds a second model to the robot’s single accelerator. In our deployed semi-autonomous stack ValueFormer runs as a standalone inference server queried by the policy client at 22 Hz (its training stride), while the π0.5 _0.5 policy runs in a separate server; both share one RTX 5090 (32 GB). The monitor is advisory and fail-safe (a missed or late tick only delays an operator alert and never enters the control path), so the question is not the critic’s own latency but whether its GPU demand degrades the policy it is meant to watch. It does, by enough to matter, and the fix needs no retraining. Where the cost comes from. A single critic tick is dominated by the frozen encoder: the six views of Section IV-D are six DINOv3 ViT-L/16 forward passes at 518518 px (the causal-transformer head is sub-millisecond by comparison). In the original configuration, those six passes run sequentially at batch 11, each embedding is copied host-side before concatenation, and the model executes in FP32; one tick then costs ≈266≈\!266 ms in isolation, a 53%53\% duty cycle against the 500500 ms period, for a peak of only 1.31.3 GB. Memory is not the constraint; compute-time occupancy is. The contention lands on the policy’s deadline. Profiled on the rig (Table VII), π0.5 _0.5 inference is 58.658.6 ms in isolation and rock-steady; run concurrently with the 22 Hz critic load it rises to 84.384.3 ms mean with a p90p90 of 100.6100.6 ms. The autonomous loop runs at 3030 Hz with a three-step prefetch, so a background inference has a ≈100≈\!100 ms budget before the current action chunk drains; a colliding critic tick pushes the policy’s tail latency onto that deadline, the prefetch is not ready, the client blocks, and the control loop stalls, a visible hitch in motion. The interference is symmetric (under policy load the critic tick itself dilates to 300300–560560 ms and occasionally overruns its 500500 ms period, so the monitor silently drops below 22 Hz), but only the policy direction has a hard deadline. Two defaults amplify the collision: the policy server inherits JAX’s 75%75\% memory preallocation (2424 GB reserved but <7<\!7 GB used), and neither process is told which streaming multiprocessors to use. TABLE VII: Deployment profile on a single RTX 5090 (32 GB): each model alone vs. both live. The value tick is the full encode-plus-head step; π0.5 _0.5 is flow-matching action sampling (1010 steps, horizon 5050). The 3030 Hz loop’s three-step prefetch gives a ≈100≈\!100 ms budget, which the concurrent policy p90p90 reaches. Component Isolated Concurrent Notes π0.5 _0.5 infer 58.658.6 ms 84.384.3 ms p90 100.6p90\,100.6, max 102.8102.8 Value tick (FP32) 266266 ms ∼400 \!400 ms 22 Hz; 53%53\% duty (isol.) Policy GPU mem. 2424 GB — JAX prealloc; <7<\!7 used Value GPU mem. 1.31.3 GB — fits; not the limit Fix: make the tick cheap, then bound the rest. The 266266 ms is largely self-inflicted, and the biggest lever needs no retrain and no accuracy change. Stacking the six views into one batched forward pass and keeping the embeddings on-GPU (removing the per-view host copies) cuts the encode 2.0×2.0×; adding a bf16 autocast on the frozen encoder takes it to 4.1×4.1× overall (Table VIII), dropping the duty cycle from 70%70\% to 17%17\% and pulling the policy’s tail latency back off the deadline. TF32 alone (2.7×2.7×) is the zero-risk floor: it changes no code path and no output semantics. The bf16 autocast changes the frozen-encoder numerics, so we gate it on the detection re-check below. Two orthogonal mitigations bound the residual interference without touching the model: pin the policy server to a memory fraction it actually uses, and cap the critic’s compute share via CUDA MPS, so a critic burst can never starve the policy; dropping the query rate to 11–1.51.5 Hz halves the collision probability at the cost of a proportionally slower alert. TABLE VIII: Value-encoder optimization headroom (encode path only, same model and inputs, measured back-to-back). Absolute times are inflated ∼1.3× \!1.3× because the policy server was resident during this run; the relative speedups are the takeaway. The transformer head adds only a few ms, so the projected optimized full tick is well under the 100100 ms budget. Encode variant Mean Speedup Duty @ 22 Hz Original: batch-1, FP32, host copies 349.8349.8 ms 1.0×1.0× 70%70\% Batched ×6× 6, FP32, on-GPU 172.0172.0 ms 2.0×2.0× 34%34\% Batched ×6× 6, TF32 129.9129.9 ms 2.7×2.7× 26%26\% Batched ×6× 6, bf16 autocast 85.185.1 ms 4.1×4.1× %17\% Applied and verified. The batched on-GPU encode and TF32 are enabled by default in the ValueFormer server, with bf16 an opt-in flag. The batched FP32 output is numerically identical to the per-view path (max|Δ|=3.5×10−5 | |=3.5× 10^-5), and bf16 shifts the value output by <10−3<\!10^-3 (VbinV_bin unchanged to four decimals). Measured on the real serving step, the per-tick cost drops from 266266 ms to ≈88≈\!88 ms (batched ++ TF32) and ≈52≈\!52 ms (++ bf16), a 33–5×5× reduction that puts the p90p90 back under the 100100 ms prefetch deadline. The bf16 encoder is safe at the deployed operating point. We validated the bf16 encoder by re-encoding the 2424-episode held-out HIL set of Section VI-E in bf16 and re-running the 22-seed ensemble detector at a matched 5%5\% false-positive rate against the FP32 reference. The bf16 embeddings stay cosine-aligned with FP32 (mean 0.999940.99994, min 0.999690.99969); frame AUROC moves 0.943→0.9420.943→ 0.942 and average precision 0.818→0.8160.818→ 0.816, and the median detection lead (2.42.4 s) is unchanged. Detection at the deployed operating point is unchanged; only a more sensitive trigger loses one of twenty onset episodes (95%→90%95\%→ 90\%) at +0.2+0.2 false alarms per minute. The server therefore defaults to bf16, while the offline feature build of Section V-B stays FP32. VII Discussion (1) The label dominates the architecture. The dominant design choice in the whole ValueFormer pipeline is the label-generation step, not the transformer. Across the five fail-episode shapes ablated in Section VI-D, the spread in validation BCE loss is roughly 4×4×, while no architectural tuning we tried (layer count, head count, hidden size, sampler weighting) produced a comparable change. MC-smooth is the only shape that avoids both an outcome-dependent leak at t=0t=0 (which scheme A introduces by scaling the whole pre-failure curve by sfail/Nss_fail/N_s) and a hard cliff at kfailk_fail (which scheme B introduces and which surfaces as high-frequency artifacts in the predicted curve). The pattern is not unique to sandwich assembly: we expect it to hold for any behavior-cloning setting where failures are a small, diversely-timed fraction of the data. Read together with Section I, the lesson is about supervision density rather than about our particular curve. Moving from one terminal outcome bit to a per-frame target is what makes the signal usable at all, but density buys nothing on its own: once the label is dense, its shape becomes the dominant free parameter, and a dense label with the wrong shape is fitted just as faithfully as a good one. Sparse supervision fails by saying too little, and badly shaped dense supervision fails by confidently saying something false. (2) Mistakes should bend the value, not break it. The intuition behind preferring smooth decay over a cliff is that a real robot’s mistakes are rarely instantaneous: a stuck grasp, a slipping ingredient, or a wrong-stage scratch usually accumulates over several seconds of policy time before the episode is unrecoverable. A smooth post-failure decay matches that physical timescale, while preserving partial credit for the stages that were genuinely completed. The cliff target (B) asks the value head to fit a step function over what is, optically, a continuous transition; the visible high-frequency artifact in the B row of Figure 12 is the network’s best effort to do that. (3) Four signatures, one model. On held-out rollouts, ValueFormer reproduces the four canonical signatures that an operator wants to read off a progress plot, and does so without any post-hoc smoothing. Each failure mode is supported by only roughly 2020 to 3030 episodes, so the fact that the model reproduces them suggests it has picked up a reasonably general notion of “progress” rather than per-episode memorization. (4) A small value head, but the encoder is the serving cost. Our ValueFormer is a dmodel=256d_model=256 transformer with two layers and four heads, for a total of roughly 3.53.5 M trainable parameters (Table I). That is well under a percent of any of the modern VLAs it supervises (for reference, π0 _0 is ∼3.3 \!3.3 B parameters). The transformer head is sub-millisecond; the deployed cost is instead the six frozen DINOv3 forward passes per tick, which we profile in Section VI-H. Where the policy already runs a DINOv3 frontend, the two could in principle share it, but in our deployed stack the π0.5 _0.5 policy and the critic run in separate servers and the critic pays for its own encode; a batched-encoder ++ bf16 path brings a tick to ≈52≈\!52 ms, which keeps the critic live at 22 Hz on the same GPU as the policy. The expensive part of the offline pipeline remains the data, not the model. (5) The causal window introduces a post-failure lag. Because the transformer is causal over a fixed history (we use 88 s in the dual-head main configuration of Table I, and used 1616 s in the earlier single-head experiments of Section VI-D), failures register in VmcV_mc several seconds behind the MC-label transition. Our visual estimate from held-out episodes is on the order of 55 to 1010 s for the 1616 s window and roughly half that for the 88 s window. Shrinking the window reduces this lag but, in our early single-head experiments, made VmcV_mc predictions noisier; one of the motivations for the dual head is to let VbinV_bin absorb the sharp-transition responsibility so that VmcV_mc does not have to compromise. Operators tolerate a lag of this order because the alternative is a person watching a screen; the cost balance would flip if we targeted very short-horizon tasks, where an asymmetric attention kernel favoring recent frames would be a reasonable next step. (6) Smooth and sharp targets are different optimization problems that share a backbone. The natural design temptation is to ask one head to do both jobs: dip on retries and snap on outright failures. We tried this in single-head form and it did not work: whatever γ and label shape we chose, the head was always either too sharp (no retry dip visible) or too smooth (no actionable abort moment). Separating the targets, while keeping the backbone shared, recovers both regimes from one forward pass at negligible cost: a few percent more parameters than single-head, and no measurable increase in wall-clock time. The same pattern likely applies to any task where the supervisor needs both a continuous progress signal and a hard yes/no for the safety filter, and the practical recipe (smooth MC on a wider value head, sharp per-frame binary on a thinner detection head, BCE on both, equal loss weight) is, we suspect, more generally useful than the specific labeling-scheme study that motivated it. VIII Limitations and Future Work In this section we summarize the limitations encountered during the development of ValueFormer and the follow-up work they motivate. Four points stand out: (i) predictions lag the true failure by several seconds; (i) failure-stage annotation is still manual; (i) closed-loop integration is now evaluated for critic-derived training weights, but the completion gain is within noise at n=20n=20 and the other closed-loop paths remain untested; and (iv) the failure taxonomy is coarse. Causal-window lag. Failures propagate into V(s)V(s) several seconds after the MC-label transition (qualitatively 55–1010 s), a visible artifact of the causal history window (3232 frames / 1616 s in the single-head runs shown; 1616 frames / 88 s in the dual-head main configuration, roughly halving the lag). An asymmetric attention kernel or a mixture-of-windows that emphasizes recent frames is a plausible remedy. Manual failure timing does not scale. Annotating 8888 failures by hand was tractable here, but a fleet-scale post-training loop produces 10310^3–10410^4 failures per week. Section IV-C takes the first step: human-in-the-loop intervention flags supply the same mistake segments automatically, and Section VI-E shows they roughly double held-out detection quality over the manual annotations alone. A stronger backbone with visual few-shot exemplars, or a small task-specific stage-completion classifier bootstrapped from the manual annotations, remains a route to labeling the residual failures that never triggered a takeover. Closed-loop integration: one path evaluated, gains within noise. The paper now closes the loop for one deployment path. Two critic-derived per-frame weights, the learned mask vf-mask and the advantage weight vf-awr, have been trained into the π0.5 _0.5 policy and evaluated on the physical station (Section VI-G); both improve subtask quality and eliminate the repeat-pick failure mode against a same-data flag-only baseline. But at n=20n=20 sandwiches the completion gain (70%→85%70\%→ 85\%) sits within noise (Fisher p=0.45p=0.45), so a measurable policy-success-rate gain is corroborated by two quasi-independent channels (the repeat-pick drop and the agreement of the two different critic weights) but not yet established; a paired re-evaluation with shared initial conditions and a McNemar test is the power fix. The remaining paths are untested end-to-end: (i) an abort / retry trigger driven by drops in V(s)V(s), (i) advantage conditioning, in which the same At=Vmc(t+HA)−Vmc(t)A_t=V_mc(t+H_A)-V_mc(t) is supplied as a conditioning token rather than a loss weight (the natural next rung beyond the weighting evaluated here), and (i) an offline filter that prunes low-value rollouts from the post-training set. The full integration with VLA post-training and the labeling loop remains work in progress. Failure taxonomy. Our failure modes are drawn empirically from rollout logs. A richer taxonomy (human-in-the-loop interventions, environmental shifts, hardware faults) would refine the label further: a hardware fault is not “partial credit” in the same sense as a grasping failure and arguably deserves a separate decay schedule. IX Conclusion ValueFormer is a compact causal-transformer value function that turns any π0 _0-family VLA into a semi-autonomous system by overlaying a calibrated per-frame progress signal. The technical core is not the transformer but the label: a stage-aware, success-then-decay Monte Carlo return whose pre-failure prefix matches the success curve and whose post-failure tail decays smoothly rather than dropping off a cliff. A controlled ablation against four alternative fail-episode shapes (outcome-scaled, cliff, α-linear mix, late-diverge) shows that this is the only shape that simultaneously avoids an outcome-dependent leak at t=0t=0 and a hard discontinuity at the failure frame, and it is also the strongest by validation BCE loss and rollout MAE. We extend the model to a dual-head configuration in which a second per-frame binary head VbinV_bin shares the same six-view ViT-L/16 backbone and is supervised by segment-based mistake intervals (tstart,tend)(t_start,t_end) rather than a single failure point, recovering an actionable abort signal that the smooth VmcV_mc head deliberately does not provide. The segment-based annotation route additionally extracts training signal from recovered mistakes in otherwise-successful rollouts, a regime the single-point table cannot reach. On a real-robot sandwich-assembly task with 1,4271,427 episodes and 213213k samples, the resulting model reaches MSE ≈3×10−4≈\!3× 10^-4 and MAE 0.0150.015 on the smooth head, reproduces the four canonical rollout signatures a supervisor needs (clean success, success-with-retry, early collapse, stuck-scratching), and trains in under three minutes on a single GPU. Two steps toward deployment follow. A batched-encoder plus bf16 inference path cuts the live per-tick serving cost 33–5×5×, so the critic can run at 22 Hz on the same GPU as the policy. An on-robot A/B shows that weighting the policy’s own post-training data with the critic improves subtask quality and removes a repeat-pick failure mode against a flag-only baseline, though the completion gain remains within noise at n=20n=20. The real prize, we believe, is the broader design pattern: a small dual-output value head whose two targets are engineered separately (smooth for the critic, sharp for the safety filter) but share one backbone and one forward pass. A step, we hope, toward robots that know when they are failing before a human has to say so. Acknowledgements The author thanks the Chef Robotics team and the open-source OpenPI, LeRobot, and DINOv3 communities, whose artifacts made this work possible. This work is supported by Chef Robotics’ Sandi project. References [1] K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, L. X. Shi, J. Tanner, Q. Vuong, A. Walling, H. Wang, and U. Zhilinsky, “π0 _0: A vision-language-action flow model for general robot control,” arXiv preprint arXiv:2410.24164, 2024. [2] K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, M. Y. Galliker, D. Ghosh, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, D. LeBlanc, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, A. Z. Ren, L. X. Shi, L. Smith, J. T. Springenberg, K. Stachowicz, J. Tanner, Q. Vuong, H. Walke, A. Walling, H. Wang, L. Yu, and U. Zhilinsky, “π0.5 _0.5: A vision-language-action model with open-world generalization,” arXiv preprint arXiv:2504.16054, 2025. [3] A. Amin, R. Aniceto, A. Balakrishna, K. Black, K. Conley, G. Connors, J. Darpinian, K. Dhabalia, J. DiCarlo, D. Driess, M. Equi, Y. Fang, C. Finn, C. Glossop, T. Godden, I. Goryachev, L. Groom, H. Hancock, K. Hausman, G. Hussein, B. Ichter, S. Jakubczak, R. Jen, T. Jones, B. Katz, L. Ke, C. Kuchi, M. Lamb, D. LeBlanc, Y. Lu, V. Mano, M. Mothukuri, K. Pertsch, A. Z. Ren, C. Rez, L. X. Shi, L. Smith, J. T. Springenberg, K. Stachowicz, A. Swerdlow, J. Tanner, M. Torne, Q. Vuong, A. Walling, H. Wang, B. Williams, L. Yu, U. Zhilinsky, and Z. Zhou, “π0.6∗ _0.6^*: A VLA that learns from experience,” arXiv preprint arXiv:2511.14759, 2025. [4] M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi et al., “OpenVLA: An open-source vision-language-action model,” arXiv preprint arXiv:2406.09246, 2024. [5] S. Liu, L. Wu, B. Li, H. Tan, H. Chen, Z. Wang, K. Xu, H. Su, and J. Zhu, “RDT-1B: A diffusion foundation model for bimanual manipulation,” arXiv preprint arXiv:2410.07864, 2024. [6] GigaAI, “GigaBrain-0.5M∗: A VLA that learns from world model-based reinforcement learning,” arXiv preprint arXiv:2602.12099, 2026. [7] R. Cadene, S. Alibert, A. Soare, Q. Gallouedec, A. Zouitine, and T. Wolf, “LeRobot: State-of-the-art machine learning for real-world robotics in PyTorch,” https://github.com/huggingface/lerobot, 2024. [8] O. Siméoni, H. V. Vo, M. Seitzer et al., “DINOv3,” arXiv preprint arXiv:2508.10104, 2025. [9] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu et al., “RT-1: Robotics transformer for real-world control at scale,” arXiv preprint arXiv:2212.06817, 2022. [10] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn et al., “RT-2: Vision-language-action models transfer web knowledge to robotic control,” arXiv preprint arXiv:2307.15818, 2023. [11] Physical Intelligence, “OpenPI: An open-source implementation of the π0 _0 vision-language-action model,” https://github.com/Physical-Intelligence/openpi, 2024. [12] Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le, “Flow matching for generative modeling,” in International Conference on Learning Representations (ICLR), 2023. [13] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song, “Diffusion policy: Visuomotor policy learning via action diffusion,” in Robotics: Science and Systems (RSS), 2023. [14] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn, “Learning fine-grained bimanual manipulation with low-cost hardware,” in Robotics: Science and Systems (RSS), 2023. [15] Xiaomi Robotics, “Xiaomi-Robotics-0: An open-sourced vision-language-action model with real-time execution,” arXiv preprint arXiv:2602.12684, 2026. [16] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski et al., “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, p. 529–533, 2015. [17] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” arXiv preprint arXiv:1509.02971, 2015. [18] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” in International Conference on Machine Learning (ICML), 2018. [19] Y. Chebotar, Q. Vuong, A. Irpan, K. Hausman, F. Xia, Y. Lu, A. Kumar, T. Yu, A. Herzog, K. Pertsch et al., “Q-Transformer: Scalable offline reinforcement learning via autoregressive Q-Functions,” in Conference on Robot Learning (CoRL), 2023. [20] I. Kostrikov, A. Nair, and S. Levine, “Offline reinforcement learning with implicit Q-Learning,” in International Conference on Learning Representations (ICLR), 2022. [21] A. Kumar, A. Zhou, G. Tucker, and S. Levine, “Conservative Q-Learning for offline reinforcement learning,” in Advances in Neural Information Processing Systems (NeurIPS), 2020. [22] Y. J. Ma, S. Sodhani, D. Jayaraman, O. Bastani, V. Kumar, and A. Zhang, “VIP: Towards universal visual reward and representation via value-implicit pre-training,” in International Conference on Learning Representations (ICLR), 2023. [23] Y. J. Ma, V. Kumar, A. Zhang, O. Bastani, and D. Jayaraman, “LIV: Language-image representations and rewards for robotic control,” in International Conference on Machine Learning (ICML), 2023. [24] B. Eysenbach, T. Zhang, S. Levine, and R. Salakhutdinov, “Contrastive learning as goal-conditioned reinforcement learning,” in Advances in Neural Information Processing Systems (NeurIPS), 2022. [25] Y. J. Ma, J. Yan, D. Jayaraman, and O. Bastani, “GoFAR: Offline goal-conditioned reinforcement learning via state-occupancy matching,” in Advances in Neural Information Processing Systems (NeurIPS), 2022. [26] A. Z. Ren, J. Lidard, L. L. Ankile, A. Simeonov, P. Agrawal, A. Majumdar, B. Burchfiel, H. Dai, and M. Simchowitz, “Diffusion policy policy optimization,” arXiv preprint arXiv:2409.00588, 2024. [27] S. Zhai et al., “VLAC: A vision-language-action-critic model for robotic real-world reinforcement learning,” arXiv preprint arXiv:2509.15937, 2025. [28] Q. Gu et al., “SAFE: Multitask failure detection for vision-language-action models,” in Advances in Neural Information Processing Systems (NeurIPS), 2025, arXiv:2506.09937. [29] J. Duan et al., “AHA: A vision-language model for detecting and reasoning over failures in robotic manipulation,” in International Conference on Learning Representations (ICLR), 2025, arXiv:2410.00371. [30] I-FailSense Authors, “I-FailSense: General robotic failure detection with vision-language models,” arXiv preprint arXiv:2509.16072, 2025. [31] StepEval Authors, “Score the steps, not just the goal: VLM-based subgoal evaluation for long-horizon manipulation,” arXiv preprint arXiv:2509.19524, 2025. [32] S. A. Sontakke, J. Zhang, S. M. R. Arnold, K. Pertsch, E. Biyik, D. Sadigh, C. Finn, and L. Itti, “RoboCLIP: One demonstration is enough to learn robot policies,” arXiv preprint arXiv:2310.07899, 2023. [33] Y. J. Ma, W. Liang, G. Wang, D.-A. Huang, O. Bastani, D. Jayaraman, Y. Zhu, L. Fan, and A. Anandkumar, “Eureka: Human-level reward design via coding large language models,” in International Conference on Learning Representations (ICLR), 2024. [34] Y. Gao, S. S. Vedula, C. E. Reiley, N. Ahmidi, B. Varadarajan, H. C. Lin, L. Tao, L. Zappella, B. Béjar, D. D. Yuh, C. C. G. Chen, R. Vidal, S. Khudanpur, and G. D. Hager, “JHU-ISI gesture and skill assessment working set (JIGSAWS): A surgical activity dataset for human motion modeling,” in Modeling and Monitoring of Computer Assisted Interventions (M2CAI), MICCAI Workshop, 2014. [35] A. P. Twinanda, S. Shehata, D. Mutter, J. Marescaux, M. de Mathelin, and N. Padoy, “EndoNet: A deep architecture for recognition tasks on laparoscopic videos,” IEEE Transactions on Medical Imaging, vol. 36, no. 1, p. 86–97, 2017. [36] Y. A. Farha and J. Gall, “MS-TCN: Multi-stage temporal convolutional network for action segmentation,” in CVPR, 2019. [37] F. Yi, H. Wen, and T. Jiang, “ASFormer: Transformer for action segmentation,” in BMVC, 2021. [38] H. Idrees, A. R. Zamir, Y.-G. Jiang, A. Gorban, I. Laptev, R. Sukthankar, and M. Shah, “The THUMOS challenge on action recognition for videos “in the wild”,” Computer Vision and Image Understanding, vol. 155, p. 1–23, 2017. [39] F. Caba Heilbron, V. Escorcia, B. Ghanem, and J. Carlos Niebles, “ActivityNet: A large-scale video benchmark for human activity understanding,” in CVPR, 2015. [40] D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price, and M. Wray, “Scaling egocentric vision: The EPIC-KITCHENS dataset,” in ECCV, 2018. [41] C.-L. Zhang, J. Wu, and Y. Li, “ActionFormer: Localizing moments of actions with transformers,” in ECCV, 2022. [42] H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe, “Let’s verify step by step,” in ICLR, 2024. [43] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta, “R3M: A universal visual representation for robot manipulation,” in CoRL, 2022. [44] J. Oh, S. Singh, and H. Lee, “Value prediction network,” in NeurIPS, 2017. [45] L. X. Shi, B. Ichter, M. Equi, L. Ke, K. Pertsch, Q. Vuong, J. Tanner, A. Walling, H. Wang, N. Fusai, A. Li-Bell, D. Driess, L. Groom, S. Levine, and C. Finn, “Hi Robot: Open-ended instruction following with hierarchical vision-language-action models,” in Proceedings of the 42nd International Conference on Machine Learning (ICML), 2025. [46] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog et al., “Do as i can, not as i say: Grounding language in robotic affordances,” arXiv preprint arXiv:2204.01691, 2022. [47] W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, A. Zeng, J. Tompson, I. Mordatch, Y. Chebotar et al., “Inner monologue: Embodied reasoning through planning with language models,” in Conference on Robot Learning (CoRL), 2022. [48] C. Lynch, A. Wahid, J. Tompson, T. Ding, J. Betker, R. Baruch, T. Armstrong, and P. Florence, “Interactive language: Talking to robots in real time,” in IEEE Robotics and Automation Letters, 2023. [49] O. Mees, L. Hermann, and W. Burgard, “What matters in language-conditioned robotic imitation learning over unstructured data,” IEEE Robotics and Automation Letters, 2022. [50] M. Kelly, C. Sidrane, K. Driggs-Campbell, and M. J. Kochenderfer, “HG-DAgger: Interactive imitation learning with human experts,” in IEEE International Conference on Robotics and Automation (ICRA), 2019.