Paper deep dive

TacSushi: Tactile-Grounded World-Action Modeling for Dexterous Sushi Manipulation

Haodi Hu, Kaen Kogashi, Toshiaki Koike-Akino

Year: 2026Venue: arXiv preprintArea: cs.ROType: PreprintEmbeddings: 48

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%

Last extracted: 9/18/2026, 4:28:46 AM

Summary

The paper introduces TacSushi, a tactile-grounded world-action policy based on the Cosmos3 architecture designed for dexterous sushi manipulation. It addresses challenges like deformation, occlusion, and uncertain contact by using feature-wise gated fusion to incorporate fingertip tactile data into the action representation. A key innovation is the use of training-only consequence supervision, where a decoder predicts future visual, tactile, and task outcomes from logged data to regularize the policy latent, without using these predictions during deployment. Experiments on real-robot trials demonstrate that TacSushi significantly outperforms baselines without tactile fusion or consequence supervision, achieving 68.3% in-distribution success and 37.5% out-of-distribution success.

Entities (8)

TacSushi · modelpolicy · 100%Consequence Supervision · training-strategy · 95%Cosmos3 · architecturebackbone · 95%Feature-wise Gated Fusion · methodtechnique · 95%Shadow Dexterous Hand Lite · robothardware · 90%Shadow Tactile Fingertip (STF) · sensor · 90%Sushi Manipulation · task · 90%GR00T · baseline-model · 85%

Relation Signals (7)

TacSushi → employs → Feature-wise Gated Fusion

confidence 95% · feature-wise gated fusion incorporates fingertip tactile features into the action representation

TacSushi → usesbackbone → Cosmos3

confidence 95% · We present TacSushi, a tactile-grounded, Cosmos3-based world-action policy

TacSushi → usestrainingstrategy → Consequence Supervision

confidence 95% · Full TacSushi achieves 68.3% average in-distribution success... compared with 36.7%... without future-consequence supervision

TacSushi → appliedto → Sushi Manipulation

confidence 90% · TacSushi: Tactile-Grounded World-Action Modeling for Dexterous Sushi Manipulation

TacSushi → controls → Shadow Dexterous Hand Lite

confidence 90% · TacSushi controls a four-finger Shadow Dexterous Hand Lite

Shadow Dexterous Hand Lite → equippedwith → Shadow Tactile Fingertip (STF)

confidence 90% · with a Shadow Tactile Fingertip (STF) sensor on each fingertip

TacSushi → outperforms → GR00T

confidence 85% · Full TacSushi achieves 68.3% average in-distribution success... compared with... M1 is GR00T without tactile input.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Dexterous food manipulation requires control under deformation, occlusion, and uncertain contact. We present TacSushi, a tactile-grounded, Cosmos3-based world-action policy that learns from recorded future consequences while acting on current observations. The backbone encodes current RGB, language, and hand state, and feature-wise gated fusion incorporates fingertip tactile features into the action representation. During training, a decoder conditioned on demonstrated action chunks predicts logged future visual observations, task progress, relative contact risk, and tactile summaries; this decoder is removed at deployment. Failed trials provide consequence supervision, but their actions are excluded from imitation. We train TacSushi on 340 successful and 50 failed real-robot trials and compare six methods in 600 separate rollouts across three in-distribution tasks and two out-of-distribution ingredient variants. To assess food quality beyond a single geometric threshold, we score terminal outcomes using an anchored visual-quality protocol that equally weights five human ratings and three vision-language-model ratings per rollout. Full TacSushi achieves 68.3% average in-distribution success and 37.5% out-of-distribution success, compared with 36.7%/10.0% without future-consequence supervision and 25.0%/17.5% with direct tactile concatenation in place of gated fusion. These comparisons support complementary benefits of feature-wise gated tactile fusion and training-only predictive supervision.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

44,404 characters extracted from source content.

Expand or collapse full text

TacSushi: Tactile-Grounded World–Action Modeling for Dexterous Sushi Manipulation Haodi Hu† Email: kaenkogashi@gmail.com Affiliation: University of Southern California, USA Affiliation: Mitsubishi Electric Research Laboratories (MERL), USA Kaen Kogashi† Affiliation: Mitsubishi Electric Research Laboratories (MERL), USA Affiliation: Mitsubishi Electric, Japan Toshiaki Koike-Akino† equal contribution Affiliation: Mitsubishi Electric Research Laboratories (MERL), USA Abstract Dexterous food manipulation requires control under deformation, occlusion, and uncertain contact. We present TacSushi, a tactile-grounded, Cosmos3-based world–action policy that learns from recorded future consequences while acting on current observations. The backbone encodes current RGB, language, and hand state, and feature-wise gated fusion incorporates fingertip tactile features into the action representation. During training, a decoder conditioned on demonstrated action chunks predicts logged future visual observations, task progress, relative contact risk, and tactile summaries; this decoder is removed at deployment. Failed trials provide consequence supervision, but their actions are excluded from imitation. We train TacSushi on 340 successful and 50 failed real-robot trials and compare six methods in 600 separate rollouts across three in-distribution tasks and two out-of-distribution ingredient variants. To assess food quality beyond a single geometric threshold, we score terminal outcomes using an anchored visual-quality protocol that equally weights five human ratings and three vision-language-model ratings per rollout. Full TacSushi achieves 68.3% average in-distribution success and 37.5% out-of-distribution success, compared with 36.7%/10.0% without future-consequence supervision and 25.0%/17.5% with direct tactile concatenation in place of gated fusion. These comparisons support complementary benefits of feature-wise gated tactile fusion and training-only predictive supervision. I Introduction Food manipulation exposes a central limitation of vision-dominant robot policies: an object can appear acceptable while local contact is already becoming unstable. Sticky rice changes shape and adheres to the fingers, seaweed bends and slips, and fragile toppings can be displaced or damaged by small errors in force or timing. Ingredient properties also vary between samples [1], and haptic cues help guide food acquisition [2]. Sushi preparation is therefore a useful test of whether a policy can relate current contact to the likely consequences of its actions, especially when the hand occludes the contact region. Research on deformable manipulation has developed simulation benchmarks, differentiable physics, and learned dynamics for cloth and elasto-plastic materials [3, 4, 5]. Learning-based granular manipulation also exploits material motion for control [6, 7]. These studies motivate learning the consequences of robot–material interaction, although their indirect manipulation through granular flow differs from direct fingertip manipulation of food. Food-handling systems have combined vision and touch for specialized acquisition or cutting operations [2, 8]. Dexterous preparation of real food remains difficult: the controller must coordinate several fingertips, tolerate adhesion and irreversible deformation, and transfer across ingredients that differ in both appearance and mechanics. This setting motivates learning contact-sensitive behavior from real demonstrations. At the policy level, generalist vision–language–action models provide reusable semantic and visual priors [9, 10, 11, 12, 13, 14], and action-chunking approaches improve temporal consistency [15, 16]. A complementary line of work learns how actions transform observations. Latent world models support planning or policy learning through imagined trajectories [17, 18], while visual-foresight methods use predicted frames in an online control loop [19, 20]. For deformable food, online model rollouts can make control depend on predictions where long-horizon geometry and contact are difficult to model. This motivates using recorded future consequences to shape the policy representation during training, without requiring predicted futures during execution. The distinction between deployed observations and privileged training signals is especially important for touch. Tactile sensing supplies local evidence that external vision may miss [21, 22] and has supported regrasp selection, grasp-outcome prediction, and state estimation under occlusion [23, 24]. Cross-modal objectives learn shared visual–tactile representations [25, 26], while predictive tactile models have enabled model-predictive control and action-conditioned slip forecasting [27, 28]. Recent imitation policies use current vision and touch, as in TACT [29], or add future-tactile prediction, as in ViTacFormer [30]. Together, these results suggest two complementary roles for tactile information, but do not fully resolve how current touch, which can guide the deployed policy, should interact with future visual and contact outcomes that exist only in the training record. TacSushi addresses this question by separating training-time consequence prediction from deployment-time control in a Cosmos3-based policy [31]. A feature-wise gate injects current fingertip features into the visual–language–proprioceptive latent that drives the action head. The same fused latent also supports a training-only consequence decoder. Current touch can therefore modify the command directly, while future observations supervise features that help predict task evolution. The deployed controller does not consume predicted outcomes. Our contribution lies in coupling this deployed gated-fusion path with training-only, multi-target consequence supervision, rather than in tactile fusion, future prediction, or action conditioning individually. The consequence decoder receives the demonstrated action chunk aligned with each training window. Failed trajectories retain their logged visual, contact, and outcome targets, while an action-loss mask excludes failed commands from imitation. These logged action–consequence pairs supervise the shared representation; the decoder is not used as a causal simulator or online planner. Our experiments address two questions: whether feature-wise gated tactile fusion improves control over direct tactile concatenation, and whether future-consequence supervision improves the deployed policy even though the auxiliary decoder is removed at deployment. We vary both factors in a matched four-variant study and examine transfer to held-out ingredients. To assess heterogeneous food outcomes, we use a human–VLM protocol that assigns each rollout a success label from its terminal appearance. The principal contributions are: 1. a Cosmos3-based world–action policy that gives current fingertip touch a direct, feature-wise gated path to action generation while reserving future consequences for training-only representation learning; 2. a multi-target objective conditioned on demonstrated actions that learns from future visual, tactile, progress, and contact-risk outcomes, with failure-aware masking that preserves consequence supervision while excluding failed actions from imitation; and 3. a real-robot study comparing tactile-fusion strategies and privileged consequence supervision, with evaluation on held-out ingredients and a multi-rater human–VLM protocol for assessing food outcomes. I Method TacSushi learns from future consequences during training and acts on current sensing at deployment. The policy observes current RGB, language, hand state, and fingertip touch and predicts an actuator-target chunk for guarded receding-horizon execution. During training, an action-conditioned decoder regularizes the fused policy latent by predicting logged visual, tactile, and task outcomes. We use world–action modeling in this representation-learning sense: the model predicts observable consequences of the demonstrated action chunk given the current observation. At deployment, the controller in Fig. 1 executes a chunk prefix and re-observes without querying the consequence decoder. Fig. 1: Detailed architecture and train–deploy asymmetry. Current visual–language–proprioceptive and tactile latents are fused into hfh_f, which independently feeds the deployed action head and a training-only consequence decoder. The decoder combines hfh_f with the demonstrated-action embedding to predict future vision, success progress, contact risk, and a tactile summary (abbreviated “Tactile” in the box). The masked action loss excludes failed commands from imitation, while their recorded consequences remain supervised. Solid blue denotes the path used in training and deployment; dashed purple denotes privileged training supervision. Only the action path reaches the receding-horizon execution controller. I-A System, Teleoperation, and Data TacSushi controls a four-finger Shadow Dexterous Hand Lite with a Shadow Tactile Fingertip (STF) sensor on each fingertip, including the thumb. Each STF contains 17 three-axis taxels, giving 68 taxels and 204 raw scalar tactile channels across the hand. The hand has 13 actuators and 16 observed joints, including three passive joints. Each action therefore specifies 13 actuator-position targets; the passive joints are observed but are not independently commanded. The policy predicts chunks of Ha=16H_a=16 action steps, where HaH_a denotes the temporal prediction horizon, not the action dimension. A Cosmos3-Nano backbone encodes RGB, language, and the 16-joint hand state; a compact tactile encoder and learned gate incorporate current contact into the action latent. Tactile preprocessing reduces each taxel’s three-axis measurement to its Euclidean magnitude, yielding 68 physical magnitude features, and computes peaks and causal summaries from samples available at or before t, as shown in Fig. 1. Fig. 2: Platform and bare-hand collection. MediaPipe landmarks drive the tactile-equipped Shadow Hand; workspace RGB, robot state/targets, and touch are logged synchronously. For data collection, a RealSense tracking camera supplies 30 Hz operator-view RGB to MediaPipe Hand Landmarker [32]. Estimated human-hand joints are retargeted to 13 robot actuator commands using fixed gains and offsets; the three passive robot joints do not receive independent targets. At up to 10 Hz, targets are clipped, low-pass filtered (α=0.30α=0.30), step-limited to 0.10 rad, smoothly interpolated, and checked for self-collision using the hand’s joint configuration. This bare-hand interface lets an operator demonstrate contact-rich motions without an exoskeleton or instrumented glove. Figure 2 shows the interface and collection setup. Separate front-view and top-view cameras record the workspace while the 16-joint robot state, 13-D actuator targets, and fingertip tactile measurements are logged synchronously at 30 Hz. The operator-view stream supports teleoperation but is excluded from policy observations. Table I summarizes the corpus. Successful trials supply action and positive-progress targets. Failed teleoperation supplies recorded adverse visual and contact consequences and a zero success-outcome target, but its action-imitation mask is zero. TABLE I: Training-corpus statistics; duration and frames per workspace-camera stream are averaged over successful trials. Statistic Rice ball Seaweed Surf clam Overall Successful trials 114 113 113 340 Failed trials 16 17 17 50 Avg. duration (s) 74 117 71 87.3 Avg. RGB frames 2,220 3,510 2,130 2,619 I-B Tactile-Gated Action and Training-Only Consequence Prediction Deployed action path. At control step t, the policy receives only the current observation xt=(It,ℓ,qt,τt)x_t=(I_t, ,q_t, _t), comprising RGB ItI_t, language instruction ℓ , the observed joint configuration qt∈ℝ16q_t ^16 (including the three passive joints), and fingertip measurements τt _t. The Cosmos3-Nano shared Mixture-of-Transformers [31] encodes the first three inputs and returns the current-conditioned action-token representation hv=fθ(It,ℓ,qt)h_v=f_θ(I_t, ,q_t); hvh_v is distinct from the future-vision decoder output. In parallel, a compact tactile encoder uses measurements available at t to produce hτ=Eψ(τt)h_τ=E_ψ( _t). The two representations are combined by the feature-wise gate in Fig. 1: gt g_t =σ⁡(Wg[hv;hτ]), =σ\! (W_g[h_v;h_τ] ), (1) hf h_f =LN⁡(hv+gt⊙Wτhτ). =LN\! (h_v+g_t W_τh_τ ). Here, WτhτW_τh_τ projects touch into the backbone feature space, and gt∈[0,1]dg_t∈[0,1]^d scales each feature of the tactile residual, where d is the hidden dimension. The action head predicts ^t=Aϕ(hf)∈ℝHa×13 a_t=A_φ(h_f) ^H_a× 13, a 16-step chunk with one 13-D actuator-target vector per step. The controller executes a prefix before re-observing. This path supplies all deployed action predictions. Training-only consequence path. During training, the fused state hfh_f also branches to a separate consequence decoder. An action encoder embeds the demonstrated actuator-target chunk t⋆∈ℝHa×13a_t ^H_a× 13 together with the current 16-joint observation as ea,t⋆=Ea(t⋆,qt)e_a,t =E_a(a_t ,q_t). Conditioned on both the current fused state and the action that was actually demonstrated, the decoder produces (I^t,v^t,r^t,b^t)=Uω(hf,ea,t⋆)( I_t, v_t, r_t, b_t)=U_ω(h_f,e_a,t ), corresponding to future vision, success progress, relative contact risk, and a tactile summary. The future-vision output uses the Cosmos generation objective; the remaining readouts regress their respective targets. Decoder losses back-propagate through hfh_f and the selected backbone adapters, so the action and consequence objectives shape a shared tactile-grounded representation. The action head does not depend on consequence-decoder outputs: neither ea,t⋆e_a,t nor a decoder prediction is passed to AϕA_φ. The action encoder and consequence decoder are removed at deployment. Aligned training windows. Each example is anchored at time t, with actions and future observations aligned over the same physical interval. It contains the current observation xtx_t, demonstrated action chunk t⋆a_t , future RGB sequence It⋆=It+1:t+HII_t =I_t+1:t+H_I, progress target ytvaluey_t value, contact-risk target ytrisky_t risk, and tactile/phase target bttacb_t tac. Actuator targets are resampled to Ha=16H_a=16 action steps and RGB to HI=33H_I=33 frames with aligned endpoints. The decoder therefore predicts the consequences of the action interval on which it is conditioned, despite the different sampling rates. Future observations and annotations serve only as training targets. The four targets describe complementary consequences. Future RGB captures visible motion and deformation; the tactile summary records contact evolution and brief loading events; progress indicates task advancement; and contact risk marks adverse interaction. None is treated as a sensor or policy input at test time. Instead, their gradients encourage hfh_f to encode outcome-relevant aspects of the current observation. Failure-aware objective. The demonstrated chunk conditions the decoder on the command associated with each logged consequence window. For failed windows, that chunk remains valid conditioning for the recorded adverse consequences, but its imitation mask is zero. Thus failed trajectories supervise the consequence objectives without treating failed commands as policy targets. The decoder is trained only on logged action–consequence pairs and is not used for counterfactual prediction. Formally, ℒtrain=mtℒaction+λvisℒfuture-vis+ℒaux,L_ train=m_tL_ action+ _ visL_ future-vis+L_ aux, (2) where mt=1m_t=1 for successful demonstrations and 00 for failed demonstrations. Only the imitation term is masked. ℒauxL_ aux contains the progress, risk, tactile-summary, and risk–summary consistency losses. We train the task-specific action, tactile, fusion, and consequence modules together with selected backbone adapters, while keeping most pretrained parameters frozen. I-C Privileged Tactile and Outcome Targets For a window beginning at sample t, let smag∈ℝK τ_s mag ^K denote the tactile-magnitude vector at sample s, where K=68K=68 is the number of per-taxel magnitude channels used for training (17 per fingertip across four STF sensors). Each channel is the Euclidean magnitude of one taxel’s three-axis measurement. For a consequence horizon of HcH_c tactile samples, define t=t+1,…,t+HcS_t=\t+1,…,t+H_c\ and summarize the strongest instantaneous contact by us=maxk∈1,…,K⁡τs,kmagu_s= _k∈\1,…,K\ _s,k mag. If the complete episode contains T samples, its normalized window endpoint is ρt=min⁡(t+Hc)/T,1 _t= \(t+H_c)/T,1\. The episode-level quantile Qα(u1:T)Q_α(u_1:T) supplies a relative contact scale, where α∈(0,1)α∈(0,1) is the selected quantile level. With a small ε>0 >0 preventing division by zero, the relative peak-contact target and risk-adjusted progress target are ytrisk y_t risk =clip(maxs∈t⁡usQα(u1:T)+ε,0,1), =clip\! ( _s _tu_sQ_α(u_1:T)+ ,0,1 ), ytvalue y_t value =mtρt(1−ytrisk). =m_t _t(1-y_t risk). (3) Here, clip⁡(⋅,0,1)clip(·,0,1) bounds the peak ratio, and mt=1m_t=1 for a successful demonstration and 00 for a failed one, matching the imitation mask in Eq. (2). In our experiments, Hc=33H_c=33 samples (1.10 s at 30 Hz), α=0.90α=0.90, and ε=10−6 =10^-6; the tactile and RGB consequence windows share the same endpoint. Thus ytrisky_t risk measures peak contact relative to the episode’s contact distribution, while ytvaluey_t value weights normalized task phase by successful outcome and relative contact risk. The tactile target retains more information than either scalar: ¯tmag τ_t mag =1Hc∑s∈tsmag, = 1H_c _s _t τ_s mag, (4) bttac b_t tac =[t+Hcmag;¯tmag;ytrisk;ρt]∈ℝ2K+2. = [ τ_t+H_c mag; τ_t mag;y_t risk; _t ] ^2K+2. Its first K entries describe contact at the window endpoint, the next K describe mean contact over the interval, and the final two record relative peak contact and normalized episode phase. With K=68K=68, this gives a 138-D target. Endpoint and mean contact distinguish residual, sustained, and transient loading that a single peak cannot represent. With [b^t]risk[ b_t]_ risk denoting the predicted risk coordinate of the summary, the nonvisual auxiliary loss is ℒaux= _ aux= λvalue(v^t−ytvalue)2+λrisk(r^t−ytrisk)2 _ value( v_t-y_t value)^2+ _ risk( r_t-y_t risk)^2 (5) +λtac2K+2‖b^t−bttac‖22+λcons(r^t−[b^t]risk)2. + _ tac2K+2\| b_t-b_t tac\|_2^2+ _ cons( r_t-[ b_t]_ risk)^2. The four λ coefficients are fixed loss weights shared across the controlled variants. The consistency term aligns the scalar risk prediction with the risk coordinate of the tactile summary. Dividing the summary loss by 2K+22K+2 normalizes it by the target dimension. Future samples, the episode quantile, and the episode outcome are used only to construct training targets and are unavailable to the deployed action head. Fig. 3: Full TacSushi future-vision predictions and synchronized ground truth for all five food categories. Each row pair shows predictions above ground truth. Table I reports fixed-set reconstruction errors; this readout is not used by deployed control. I-D Training, Deployment, and Evaluated Methods Training jointly optimizes all enabled terms in Eq. (2). Successful windows supervise action generation, and every valid window supervises the enabled consequence objectives. The failure mask mtm_t affects only imitation. Task-specific modules are trained together with selected Cosmos3 adapters; the remaining pretrained parameters are frozen. At deployment, current RGB, the 16-joint state, touch, and language enter the solid-blue path in Fig. 1. The server returns one Ha×13H_a× 13 action chunk. The receding-horizon controller clamps the actuator targets, checks the associated hand configuration for self-collision, executes the first Kexec≤HaK_ exec≤ H_a steps, and re-observes before requesting the next chunk, where KexecK_ exec is the number of steps committed per update. We used KexecK_ exec = 12 in this work. Controlled Cosmos3 variants (M3–M6). We vary two factors: the tactile-fusion strategy and the use of training-only consequence supervision. All four variants receive current tactile observations. M3 uses direct tactile concatenation without consequence supervision (gate−-, auxiliary−-); M4 uses the same concatenation with consequence supervision (gate−-, auxiliary++); M5 uses feature-wise gated tactile fusion without consequence supervision (gate++, auxiliary−-); and M6 is full TacSushi, with gated fusion and consequence supervision (gate++, auxiliary++). The gate−- variants retain the tactile encoder and directly concatenate its output with the visual–language–proprioceptive representation. The gate++ variants use the gated residual in Eq. (1). Removing consequence supervision sets the future-vision and all other consequence-loss weights to zero while preserving the deployed tactile pathway. M3–M6 share the training data, data order, number of optimization steps, optimizer schedule, action-loss settings, and guarded executor. M3/M4 and M5/M6 compare consequence supervision at a fixed fusion architecture; M3/M5 and M4/M6 compare gated fusion with direct concatenation at a fixed supervision setting. All four retain the pretrained visual generator for offline evaluation, but only M4 and M6 adapt it using future-consequence targets. None uses the generator during deployed control. System-level baselines (M1–M2). M1 is GR00T without tactile input. M2 augments GR00T with current touch: an MLP embeds the tactile portion of the state vector, and a sigmoid gate injects that embedding into the state representation before action prediction. M1–M2 provide cross-backbone system comparisons. Fig. 4: Representative successful executions on all five conditions. The first three rows show in-distribution rice-ball shaping, seaweed wrapping, and surf-clam placement; the final two show held-out black-rice and salmon ingredient variants. Each row begins with the desired appearance and then traces top-camera frames from the initial state to the terminal outcome. I Experiments I-A Tasks and Rollout Protocol We evaluate the six methods defined in Sec. I-D on three in-distribution (ID) tasks—rice-ball shaping, seaweed wrapping, and surf-clam placement—and two held-out out-of-distribution (OOD) ingredient variants, black rice and salmon. Each method–task condition includes 20 trials, giving 600 real-robot rollouts separate from the training corpus. Every method uses the platform and guarded execution interface in Fig. 2. Rice-ball shaping compacts loose rice into a self-supporting oval portion; seaweed wrapping closes the sheet around the rice; and surf-clam placement centers and retains the topping without visible crushing. Black rice preserves the shaping objective while changing the ingredient, whereas salmon replaces the seen surf-clam topping while preserving the placement objective. Neither held-out ingredient is used for task-specific adaptation. Figure 4 shows intermediate behavior omitted by a terminal score; its final frames illustrate the visual outcomes judged by the protocol below. TABLE I: Human and AI-ensemble visual-quality scores on the five-point scale (mean ± standard error of the mean, SEM; 20 rollouts per method–category cell). Category MAE averages the absolute human–AI differences between the six method means; lower is better. M1–M6 are defined in Sec. I-D. Category Group M1 M2 M3 M4 M5 M6 MAE (↓ ) Rice ball Human 1.96±0.291.96± 0.29 2.66±0.322.66± 0.32 1.80±0.261.80± 0.26 2.22±0.312.22± 0.31 2.02±0.292.02± 0.29 3.34±0.283.34± 0.28 0.07 AI 2.02±0.262.02± 0.26 2.65±0.352.65± 0.35 1.91±0.281.91± 0.28 2.28±0.272.28± 0.27 2.18±0.312.18± 0.31 3.31±0.263.31± 0.26 Seaweed Human 1.42±0.191.42± 0.19 2.26±0.312.26± 0.31 1.68±0.231.68± 0.23 1.68±0.231.68± 0.23 2.22±0.312.22± 0.31 2.92±0.312.92± 0.31 0.09 AI 1.62±0.181.62± 0.18 2.42±0.292.42± 0.29 1.77±0.221.77± 0.22 1.76±0.221.76± 0.22 2.28±0.292.28± 0.29 2.93±0.342.93± 0.34 Surf clam Human 1.92±0.281.92± 0.28 2.78±0.322.78± 0.32 1.46±0.191.46± 0.19 1.88±0.281.88± 0.28 2.50±0.282.50± 0.28 3.20±0.293.20± 0.29 0.10 AI 2.04±0.242.04± 0.24 2.76±0.302.76± 0.30 1.62±0.181.62± 0.18 2.06±0.262.06± 0.26 2.58±0.302.58± 0.30 3.24±0.283.24± 0.28 Salmon (OOD) Human 1.44±0.171.44± 0.17 1.58±0.231.58± 0.23 1.38±0.141.38± 0.14 1.98±0.241.98± 0.24 1.52±0.191.52± 0.19 2.26±0.342.26± 0.34 0.13 AI 1.62±0.181.62± 0.18 1.77±0.221.77± 0.22 1.47±0.131.47± 0.13 2.10±0.262.10± 0.26 1.58±0.181.58± 0.18 2.42±0.302.42± 0.30 Black rice (OOD) Human 1.30±0.161.30± 0.16 1.58±0.231.58± 0.23 1.36±0.141.36± 0.14 1.52±0.221.52± 0.22 1.54±0.191.54± 0.19 2.27±0.292.27± 0.29 0.11 AI 1.52±0.131.52± 0.13 1.78±0.221.78± 0.22 1.47±0.131.47± 0.13 1.50±0.181.50± 0.18 1.56±0.181.56± 0.18 2.18±0.292.18± 0.29 I-B Human–VLM Outcome Evaluation and User Survey Acceptable food outcomes vary with ingredient and task, making a single geometric or force threshold unsuitable for judging shape, wrapping, topping placement, and visible damage. A fixed geometric tolerance can reject natural variation yet overlook obvious failure. We therefore combine a photo-based human survey with VLM evaluation under a shared, anchored five-point visual-quality scale. Task-specific criteria include compact, self-supporting rice; a wrap that remains closed; a centered and retained topping; and no visible crushing. Survey protocol. Fifty voluntary, uncompensated participants provided written consent and evaluated terminal images from the 600 robot rollouts. Each image was entered five times into the assignment pool, producing 3,000 image-rating assignments and five human ratings per rollout. Each participant rated 60 images with 12 randomly assigned images from each of the five food categories. As a result, each category contributed 120 unique rollout images and 600 rating assignments. Before rating, participants read brief guidance on desirable sushi presentation: an oval rice portion, balanced topping placement, gentle pressing, and avoidance of over-compression. The anchored five-point scale was 5, excellent/visually perfect; 4, good/well-prepared; 3, acceptable/neutral; 2, poor/slightly messy; and 1, very poor/disorganized. The study underwent institutional operations review, with the institution withheld for double-blind review. Participation required approximately 15 minutes, and responses were anonymized. Participants were told that pictured sushi could have been produced by a chef, a conveyor-belt machine, or a dexterous robotic hand, but individual images did not display source labels. Trial-level composite score. GPT-5.6, Claude Opus 4.8, and Gemini 3.5 independently score the same terminal images using a common prompt and the survey rubric. The input to the 3 VLMs is a survey with a prompt that describes the general evaluation rule and instructions as the same as used for the human survey. Each rollout therefore receives eight equally weighted ratings: five from humans and one from each of the three VLMs. For trial j, let ℋjH_j be its set of five human raters, rj,hhumanr_j,h human the rating from human h, and rj,kAIr_j,k AI the rating from VLM k∈1,2,3k∈\1,2,3\. The human mean r¯jhuman r_j human, AI-ensemble mean r¯jAI r_j AI, and composite score sjs_j are r¯jhuman r_j human =1|ℋj|∑h∈ℋjrj,hhuman, = 1|H_j| _h _jr_j,h human, (6) r¯jAI r_j AI =13∑k=13rj,kAI, = 13 _k=1^3r_j,k AI, sj s_j =58r¯jhuman+38r¯jAI. = 58 r_j human+ 38 r_j AI. Every individual rating contributes 1/81/8 of the composite score. Consequently, the five-rater human mean carries a weight of 5/85/8 (62.5%), and the three-model AI mean carries 3/83/8 (37.5%). The weighting is equal across individual ratings, not across the two evaluator groups. Trial j is successful if and only if sj≥3s_j≥ 3, with 3 marking acceptable task completion. Each success label is computed from the eight ratings of that specific rollout, not from a method-level or category-level mean. Success rates average the binary labels over 20 trials per method–task condition. Task-level Wilson intervals use the integer success count with n=20n=20. The all-task panel reports a descriptive pooled proportion and Wilson interval for the prespecified five-task mixture (n=100n=100), rather than a common success probability for all tasks. Intervals in Fig. 5 reflect rollout sampling conditional on one training run and checkpoint per method and the fixed labels from Eq. (6). They do not capture training-seed variability or uncertainty from the evaluators and human raters. Human–AI score comparison. Table I reports the human and AI-ensemble scores separately by method and food category. Each cell gives the mean and SEM of the corresponding rollout-level means in Eq. (6) over 20 trials. The final column averages the absolute differences between the six paired human and AI method means within each category, in rating points. It therefore describes agreement between aggregate scores rather than ratings of individual images. Full TacSushi (M6) receives the highest mean rating from both evaluator groups in every category. Its human scores are 3.34±0.283.34± 0.28 for rice ball, 2.92±0.312.92± 0.31 for seaweed, and 3.20±0.293.20± 0.29 for surf clam, compared with 2.26±0.342.26± 0.34 for salmon and 2.27±0.292.27± 0.29 for black rice. Its relative advantage thus persists on the held-out ingredients, although their lower absolute ratings indicate room to improve visual quality. AI-ensemble means tend to be higher than human means, but the direction and magnitude of the difference vary by method and category. TABLE I: Offline 33-frame future-vision reconstruction diagnostic on a fixed 176-clip set (5,808 frames/method). Lower is better. The visual readout is absent from deployed control, and these errors do not by themselves isolate latent quality. Model MAE (%) ↓ 1−SSIM1-SSIM (%) ↓ LPIPS ↓ M3 11.4% 35.1% 0.29 M4 7.5% 28.9% 0.21 M5 5.6% 22.9% 0.17 M6 (TacSushi) 3.8% 17.7% 0.13 Fig. 5: Real-robot success under the trial-level human–VLM visual-outcome rule. Error bars are two-sided 95% Wilson intervals using n=20n=20 per task and n=100n=100 for the descriptive pooled panel; M1–M6 are defined in Sec. I-D. I-C Manipulation Success and Controlled Comparisons Full TacSushi achieves success rates of 75%, 60%, and 70% on rice ball, seaweed, and surf clam, respectively (68.3% ID average), and 35% and 40% on held-out black rice and salmon (37.5% OOD average). Both factors improve average ID and OOD success when the other is held fixed. Enabling consequence supervision raises ID/OOD success from 15.0/5.0% to 25.0/17.5% with direct tactile concatenation (M3→ 4), and from 36.7/10.0% to 68.3/37.5% with gated fusion (M5→ 6). Replacing direct tactile concatenation with feature-wise gated fusion raises ID/OOD success from 15.0/5.0% to 36.7/10.0% without consequence supervision (M3→ 5), and from 25.0/17.5% to 68.3/37.5% with consequence supervision (M4→ 6). These matched comparisons provide the main experimental evidence for the two design choices. M5 and M6 have identical deployed inputs and architecture, yet M6 achieves 31.7 percentage points higher ID success and 27.5 points higher OOD success. This comparison supports the benefit of training-only consequence supervision without additional deployment-time sensing or decoder computation. With consequence supervision enabled, M6 exceeds the concatenation variant M4 by 43.3 ID points and 20.0 OOD points, supporting the benefit of feature-wise gated tactile fusion over direct tactile concatenation. Since all four variants receive current touch, these comparisons assess how tactile information is fused, not whether it is available. The positive contrasts support complementary benefits of gated fusion and consequence supervision. Full TacSushi also exceeds the strongest system-level baseline, GR00T+tactile (M2), by 20.0 ID points, 22.5 OOD points, and 21 points in the descriptive five-task pool (56% versus 35%). Pooled success for M1–M6 is 15%, 35%, 11%, 22%, 26%, and 56%, respectively. The gains on black rice and salmon show transfer to the two prespecified ingredient shifts, but do not establish broad OOD generalization. Because M5 disables all consequence objectives together, its comparison with M6 attributes the benefit to their joint supervision rather than any individual prediction target. I-D Auxiliary Future-Vision Diagnostic We next examine whether consequence supervision improves future-vision reconstruction. We evaluate the inherited visual readout offline for M3–M6 on a fixed diagnostic set of 176 synchronized clips, each containing HI=33H_I=33 target frames (5,808 frames per method). The diagnostic clips are trajectory-disjoint from all training episodes. M4 and M6 adapt the readout through future-vision supervision; M3 and M5 retain it for this diagnostic but receive no future-vision loss. Decoded RGB values are normalized to [0,1][0,1]. Table I reports 100×MAE100×MAE, frame-averaged 100×(1−SSIM)100×(1-SSIM) with an 11×1111×11 Gaussian window [33], and LPIPS [34]; lower is better. Figure 3 shows qualitative examples. With the deployed gated-fusion architecture held fixed, consequence supervision reduces the three errors from 5.6%, 22.9%, and 0.17 for M5 to 3.8%, 17.7%, and 0.13 for M6. The direct-concatenation pair M3/M4 shows the same trend. These results indicate improved future-vision reconstruction alongside the control improvements reported above. The predictions in Fig. 3 preserve coarse scene evolution but remain imperfect around rice and seaweed. This diagnostic does not isolate changes in hfh_f, because the future-vision objective updates both the shared representation and the decoder. IV Discussion The controlled comparisons assess two design choices: feature-wise gated fusion versus direct tactile concatenation, and the use of joint consequence supervision. Current tactile input is available in every Cosmos3 variant, so these ablations do not measure the benefit of adding touch to a vision-only policy. They also do not separate the effects of future vision, tactile summaries, task progress, contact risk, or failed-window supervision. Target-specific and stop-gradient ablations, evaluated with a common probe on trajectory-disjoint data, would distinguish these contributions. Continual adaptation to new ingredients is a promising extension of this work. The benefit of training-only consequence supervision suggests that newly logged visual and tactile outcomes could help update contact-sensitive features without adding computation to the deployed action path. In particular, unsuccessful interactions could supply consequence targets while their commands remain excluded from imitation. A chronological adaptation study could test whether this strategy improves performance on new ingredients through fresh post-update rollouts, while replay and retention tests on the foods could assess preservation of learned skills. The present evaluation uses a fixed policy: none of the 600 evaluation rollouts is used for updates. V Conclusion We presented TacSushi, a Cosmos3-based dexterous policy that uses feature-wise gated tactile fusion for control and action-conditioned future-consequence supervision during training. Failed windows retain their observed consequence targets while their actions are excluded from imitation, and all auxiliary readouts are removed at deployment. Across 600 real-robot rollouts, full TacSushi achieved higher visual-outcome success than matched variants using direct tactile concatenation or omitting consequence supervision, including on two ingredients held out from task-specific training. Both human and AI evaluators assigned it the highest mean visual-quality score in every category, although scores were lower on held-out ingredients. These findings support combining gated tactile fusion with predictive supervision, while leaving their underlying mechanisms and consistency across training seeds for further study. References [1] Z. Wang, S. Hirai, and S. Kawamura, “Challenges and opportunities in robotic food handling: A review,” Frontiers in Robotics and AI, vol. 8, p. 789107, 2021. [2] T. Bhattacharjee, G. Lee, H. Song, and S. S. Srinivasa, “Towards robotic feeding: Role of haptics in fork-based food manipulation,” IEEE Robotics and Automation Letters, vol. 4, no. 2, p. 1485–1492, 2019. [3] X. Lin, Y. Wang, J. Olkin, and D. Held, “SoftGym: Benchmarking deep reinforcement learning for deformable object manipulation,” in Proceedings of the 2020 Conference on Robot Learning, ser. Proceedings of Machine Learning Research, vol. 155. PMLR, 2021, p. 432–448. [4] Z. Huang et al., “PlasticineLab: A soft-body manipulation benchmark with differentiable physics,” in International Conference on Learning Representations, 2021. [5] H. Shi, H. Xu, Z. Huang, Y. Li, and J. Wu, “RoboCraft: Learning to see, simulate, and shape elasto-plastic objects with graph networks,” The International Journal of Robotics Research, vol. 43, no. 4, p. 533–549, 2024. [6] H. Hu, F. Qian, and D. Seita, “Learning granular media avalanche behavior for indirectly manipulating obstacles on a granular slope,” in Proceedings of the 8th Conference on Robot Learning, ser. Proceedings of Machine Learning Research, vol. 270. PMLR, 2025, p. 3225–3241. [Online]. Available: https://proceedings.mlr.press/v270/hu25c.html [7] H. Hu, Y. Wu, D. Seita, and F. Qian, “Granular loco-manipulation: Repositioning rocks through strategic sand avalanche,” in Proceedings of the 9th Conference on Robot Learning, ser. Proceedings of Machine Learning Research, vol. 305. PMLR, 2025, p. 4566–4584. [Online]. Available: https://proceedings.mlr.press/v305/hu25d.html [8] S. Herland et al., “Sashimi-bot: Autonomous tri-manual advanced manipulation and cutting of deformable objects,” 2025. [Online]. Available: https://arxiv.org/abs/2511.11223 [9] A. Brohan et al., “RT-1: Robotics transformer for real-world control at scale,” arXiv preprint arXiv:2212.06817, 2022. [10] Open X-Embodiment Collaboration et al., “Open X-embodiment: Robotic learning datasets and RT-X models,” arXiv preprint arXiv:2310.08864, 2023. [11] A. Brohan et al., “RT-2: Vision-language-action models transfer web knowledge to robotic control,” arXiv preprint arXiv:2307.15818, 2023. [12] Octo Model Team et al., “Octo: An open-source generalist robot policy,” arXiv preprint arXiv:2405.12213, 2024. [13] M. J. Kim et al., “OpenVLA: An open-source vision-language-action model,” arXiv preprint arXiv:2406.09246, 2024. [14] NVIDIA, “Isaac GR00T N1.7: A foundation model for generalist robots,” https://github.com/NVIDIA/Isaac-GR00T, 2026, accessed: 2026-06-17. [15] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn, “Learning fine-grained bimanual manipulation with low-cost hardware,” in Robotics: Science and Systems, 2023. [16] C. Chi et al., “Diffusion policy: Visuomotor policy learning via action diffusion,” in Robotics: Science and Systems, 2023. [17] D. Hafner et al., “Learning latent dynamics for planning from pixels,” in Proceedings of the 36th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 97. PMLR, 2019, p. 2555–2565. [18] D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi, “Dream to control: Learning behaviors by latent imagination,” in International Conference on Learning Representations, 2020. [19] C. Finn and S. Levine, “Deep visual foresight for planning robot motion,” in IEEE International Conference on Robotics and Automation, 2017, p. 2786–2793. [20] F. Ebert, C. Finn, S. Dasari, A. Xie, A. X. Lee, and S. Levine, “Visual foresight: Model-based deep reinforcement learning for vision-based robotic control,” arXiv preprint arXiv:1812.00568, 2018. [21] Q. Li, O. Kroemer, Z. Su, F. F. Veiga, M. Kaboli, and H. J. Ritter, “A review of tactile information: Perception and action through touch,” IEEE Transactions on Robotics, vol. 36, no. 6, p. 1619–1634, 2020. [22] M. Lambeta et al., “DIGIT: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation,” IEEE Robotics and Automation Letters, vol. 5, no. 3, p. 3838–3845, 2020. [23] R. Calandra et al., “More than a feeling: Learning to grasp and regrasp using vision and touch,” IEEE Robotics and Automation Letters, vol. 3, no. 4, p. 3300–3307, 2018. [24] P. Sodhi, M. Kaess, M. Mukadam, and S. Anderson, “Learning tactile models for factor graph-based estimation,” in IEEE International Conference on Robotics and Automation, 2021, p. 12 586–12 592. [25] M. A. Lee et al., “Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks,” in IEEE International Conference on Robotics and Automation, 2019. [26] Y. Li, J.-Y. Zhu, R. Tedrake, and A. Torralba, “Connecting touch and vision via cross-modal prediction,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, p. 10 609–10 618. [27] S. Tian et al., “Manipulation by feel: Touch-based control with deep predictive models,” in IEEE International Conference on Robotics and Automation, 2019, p. 818–824. [28] W. Mandil, K. Nazari, and A. Ghalamzan, “Action conditioned tactile prediction: Case study on slip prediction,” in Proceedings of Robotics: Science and Systems, 2022. [29] M. Murooka et al., “TACT: Humanoid whole-body contact manipulation through deep imitation learning with tactile modality,” IEEE Robotics and Automation Letters, vol. 10, no. 8, p. 7819–7826, 2025. [30] L. Heng, H. Geng, K. Zhang, P. Abbeel, and J. Malik, “ViTacFormer: Learning cross-modal representation for visuo-tactile dexterous manipulation,” arXiv preprint arXiv:2506.15953, 2025. [31] NVIDIA, “Cosmos 3: Omnimodal world models for physical AI,” arXiv preprint arXiv:2606.02800, 2026. [Online]. Available: https://research.nvidia.com/labs/cosmos-lab/cosmos3/technical-report.pdf [32] F. Zhang et al., “MediaPipe Hands: On-device real-time hand tracking,” arXiv preprint arXiv:2006.10214, 2020. [33] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, p. 600–612, 2004. [34] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in Proc. CVPR, 2018.

Instant research discovery