Paper deep dive
Do-JEPA: From Masking to Intervention in Latent World Models
Hossein Resani, Javen Qinfeng Shi
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Latent world models are trained to predict what happens next, so nothing in their objective separates what an action caused from what merely co-occurred with it. Object-masking models such as C-JEPA intervene on what the predictor can see; we intervene on what physically happens. From one saved simulator state we run the dynamics under an action $a$ and under a reference action $a_{\varnothing}$, and train the model to predict the difference $\Delta z=z^{a}-z^{a_{\varnothing}}$ between the two latent futures. The resulting objective, Do-JEPA, has an effect loss, a support loss (where the action enters), a propagation loss (where its effect travels) and invariance losses (what must not change). In a synthetic system with object-aligned variables, support supervision finds the directly intervened object in 99.95% of test cases, where a sparse action mask sends the action to a nuisance slot in every case, and response-onset supervision recovers the ring-shaped propagation graph (edge AUROC 0.975 vs. 0.624). From pixels, the effect loss beats a control trained on exactly the same data: it lowers latent effect error by 28.4% on an end-to-end LeWM model and physical effect error by 13.5% when trained and tested on natural action sequences, and on three independently generated CausalWorld benchmarks it lowers responsive effect error by about 20% under physics shifts and the latent context sensitivity of predicted effects by 66%. Trained from scratch it costs factual accuracy; fine-tuning an existing model with it removes this cost. Together, these results show that intervening on the world, rather than on what the model sees, helps latent world models predict what their actions cause.
Tags
Links
- Source: https://arxiv.org/abs/2609.37378v1
- Canonical: https://arxiv.org/abs/2609.37378v1
Trouble viewing inline? Open PDF directly →
Full Text
120,140 characters extracted from source content.
Expand or collapse full text
Do-JEPA: From Masking to Intervention in Latent World Models Hossein Resani & Javen Qinfeng Shi Affiliation: Australian Institute for Machine Learning, Adelaide University Affiliation: Responsible AI Research Centre Abstract Latent world models are trained to predict what happens next, so nothing in their objective separates what an action caused from what merely co-occurred with it. Object-masking models such as C-JEPA intervene on what the predictor can see; we intervene on what physically happens. From one saved simulator state we run the dynamics under an action a and under a reference action a∅a_ , and train the model to predict the difference Δz=za−za∅ z=z^a-z^a_ between the two latent futures. The resulting objective, Do-JEPA, has an effect loss, a support loss (where the action enters), a propagation loss (where its effect travels) and invariance losses (what must not change). In a synthetic system with object-aligned variables, support supervision finds the directly intervened object in 99.95%99.95\% of test cases, where a sparse action mask sends the action to a nuisance slot in every case, and response-onset supervision recovers the ring-shaped propagation graph (edge AUROC 0.9750.975 vs. 0.6240.624). From pixels, the effect loss beats a control trained on exactly the same data: it lowers latent effect error by 28.4%28.4\% on an end-to-end LeWM model and physical effect error by 13.5%13.5\% when trained and tested on natural action sequences, and on three independently generated CausalWorld benchmarks it lowers responsive effect error by about 20%20\% under physics shifts and the latent context sensitivity of predicted effects by 66%66\%. Trained from scratch it costs factual accuracy; fine-tuning an existing model with it removes this cost. Together, these results show that intervening on the world, rather than on what the model sees, helps latent world models predict what their actions cause. 1 Introduction Latent world models learn a transition z^t+1=Fθ(zt,at) z_t+1=F_θ(z_t,a_t) in a learned embedding space and use it to predict and to plan (Ha and Schmidhuber, 2018; Hafner et al., 2023; Zhou et al., 2024); joint-embedding predictive architectures (JEPAs) are a common choice (LeCun, 2022; Assran et al., 2023; Assran et al., 2025; Balestriero and LeCun, 2025). A planner asks such a model a counterfactual question: what would happen if I took this action instead of that one? The model is trained on a different question: what happened next in the logged data? The two come apart when something the action does not control co-varies with the outcome, for example a hidden cause shared by the behaviour policy and the rendered scene. The model can then fit the logs by crediting the wrong variable, and prediction error does not show it. In our synthetic benchmark (Sec. 4.1), the model with the lowest prediction and effect error has no notion of where an action enters, and a model with a sparse action-entry mask sends every action into a nuisance slot. Object-centric JEPAs add structure but keep the observational target. C-JEPA (Nam et al., 2026) masks whole object slots, so that the predictor must infer a hidden object from the others. As its authors note, this intervenes on observability, not on the process that generates the data: masking an object changes what the model sees, not what the world does, and the targets still come from the logged rollout. Figure 1: Masking versus intervention. Both methods see the same history, slots and action; orange marks where each intervenes. Left: C-JEPA (Nam et al., 2026) hides an object’s slots and predicts them from the others, with targets from the same logged rollout: it changes what the model sees. Right: Do-JEPA runs one saved simulator state forward under a and under a∅a_ and supervises the difference Δz z of the two latent futures (Eq. 3): it changes what happens. Frames: Push-T. A simulator allows a stronger intervention. We save its full state and run it forward twice, once with an action a and once with a reference action a∅a_ (Fig. 1). State, context, history and noise are shared, so the difference between the two latent futures is the change caused by executing a instead of a∅a_ in that state, and nothing else. We call objectives built on such pairs Do-JEPA: C-JEPA intervenes on the observation, Do-JEPA on the world. A world model that knows what its actions do should say what an action changed (effect), where it entered (support), where its effect can travel (propagation) and what must stay the same (invariance). Do-JEPA has one loss for each, computed from the pairs alone. We test each where it can be measured: support and propagation against known ground truth, invariance with given variables, and the effect loss end to end from pixels against a control that sees exactly the same data. Our contributions are summarised as follows. First, we introduce Do-JEPA, an interventional objective for latent world models that derives four training signals from exact same-state rollout pairs: the effect of an action, where it enters, where its effect travels and what must not change. It needs no graph labels or object identities (Sec. 3). Second, we show that the direct target and the propagation graph can be recovered in controlled settings. Support supervision identifies the directly intervened object in 99.9599.95–100%100\% of test cases on three seeds, whereas a sparse mask picks the wrong slot at chance level. Onset supervision recovers the propagation graph, and the observation rate decides whether one-hop edges or only reachability can be recovered (Sec. 4.1). Third, we provide evidence from pixels against an information-matched control: paired supervision improves effect prediction on end-to-end LeWM and on three CausalWorld benchmarks against a control trained on the same data, and fine-tuning with it kept factual accuracy in our tests (Sec. 4.3). 2 Related Work Latent world models and JEPAs. JEPAs predict in embedding space (LeCun, 2022; Assran et al., 2023) and now drive action-conditioned video models and planners (Assran et al., 2025; Zhou et al., 2024). LeWM (Maes et al., 2026b) trains such a model end to end from pixels with the SIGReg regulariser (Balestriero and LeCun, 2025). We use LeWM as a backbone and, with its CEM planner (García et al., 1989; Rubinstein, 1999) unchanged, for a planning study (App. F). Interventions and causal world models. Interventions make latent causal variables identifiable where observations do not (Schölkopf et al., 2021; Lippe et al., 2022; Lippe et al., 2023; Brehmer et al., 2022; Ahuja et al., 2023; von Kügelgen et al., 2021); we share this premise but do not recover causal variables or graphs from pixels. Other work learns modular mechanisms (Lei et al., 2022) or interaction graphs (Li et al., 2020), chooses actions that separate causal hypotheses (Sontakke et al., 2021), or argues for intervention-centric evaluation (Liu et al., 2023). Closest to us, C-JEPA (Nam et al., 2026) masks object-slot trajectories; we re-implement it on the same slots. 3 Method 3.1 Setting Setup and notation. An environment has a physical state sts_t, an action at∈a_t and a context c∈c that changes the observation but not the physics: the next state is st+1=f(st,at,εt)s_t+1=f(s_t,a_t, _t), with exogenous noise εt _t, and the observation is ot=g(st,c)o_t=g(s_t,c). Examples of c are a pattern rendered into the image and a vector of nuisance variables. An encoder EψE_ψ maps an observation to K slots, zt=(zt1,…,ztK)z_t=(z^1_t,…,z^K_t) with zti∈ℝdz^i_t ^d, and a predictor FθF_θ maps a history of H latent states and an action to the next latent state: z^t+1=Fθ(zt−H+1:t,at)∈ℝK×d. z_t+1=F_θ (z_t-H+1:t,\,a_t ) ^K× d. (1) Object-slot models have one slot per object or nuisance variable (K>1K>1); the LeWM backbone has one global latent (K=1K=1, d=192d=192). A hat marks a prediction, N is the batch size, and Table 5 lists every symbol with its shape, role and availability. JEPA world models. A JEPA world model is trained to predict the latent of the next observation, ℒpred=1N∑n=1N∥z^t+1,n−zt+1,n∥22,L_pred= 1N _n=1^N \, z_t+1,n-z_t+1,n _2^2, (2) where zt+1,nz_t+1,n encodes the observed next frame of example n, together with a term that prevents collapse (an exponential-moving-average target encoder, or SIGReg in LeWM). Targets, here and below, are fixed for given variables or a frozen encoder; in our end-to-end LeWM-class models, as in LeWM, gradients also flow into them. Eq. 2 is observational, meaning that no term says which part of the observed change the action caused. Why masked prediction is not enough. C-JEPA recovers hidden slots from the others, with targets from the same logged trajectory. This teaches object interactions but does not change the data. If a nuisance c is correlated with the outcome in training and the state estimate is imperfect, a predictor can use c and fails when the correlation changes. Do-JEPA changes the data instead, adding a rollout that differs only in the action. 3.2 Paired interventions and what they reveal Paired trajectories. From a state sts_t we save the full simulator state, including contact caches and the random-number state, and run it twice for k steps with the same context and noise. The factual branch applies a; the reference branch applies a∅a_ (a physical no-op unless stated otherwise) and later actions are identical. The result is a paired trajectory that differs only in the intervention, do(a)do(a) versus do(a∅)do(a_ ); replaying a branch reproduces it exactly (maximum difference 0.00.0). The paired effect at step t+ℓt+ and its prediction are Δzt+ℓ=zat+ℓ−za∅t+ℓ,Δz^t+ℓ=z^at+ℓ−z^a∅t+ℓ,ℓ=1,…,k, z_t+ =z^a_t+ -z^a_ _t+ , z_t+ = z^\,a_t+ - z^\,a_ _t+ , =1,…,k, (3) both in ℝK×dR^K× d. The target Δz z needs both branches and exists only at training time, whereas the prediction Δz z runs the predictor twice from the same history. Because the branches share state, context and noise, Δz z is the effect of replacing a∅a_ by a in this state, not an average over states. Any latent component shared by both branches cancels: exactly for an additive context contribution, but not for one that interacts with the state. A pair is responsive if the manipulated object moves by more than about 22 px (Push-T) or 22 m (CausalWorld; App. C). Support, responses and propagation. The direct target of an intervention is the set of variables it changes at once. With object-aligned slots, slot i belongs to the direct support if its immediate response ∥Δzt+1i∥ z^\,i_t+1 is large. This label comes from the pair, not from the simulator’s graph, and it is correct only if one observation step is too short for the effect to reach a second object (temporal resolvability; Sec. 4.1). The response set R contains every slot j with ∥Δzt+ℓj∥>τ z^\,j_t+ >τ for some ℓ≤k ≤ k; its members outside the direct support are the downstream responders, and slots outside R are the invariant variables of the pair. The onset of slot j is tj=minℓ:∥Δzt+ℓj∥>τt_j= \ : z^\,j_t+ >τ\. If j starts to respond one step after i, then i is a candidate cause of j’s response, which gives a propagation label yjiedge∈0,1y^edge_ji∈\0,1\ for the message from slot i to slot j. These labels describe how this intervention propagated. Therefore, an interaction that the rollout does not use receives no positive label. Nuisance and context. A context twin keeps the state and the action and changes only the context, o′=g(s,c′)o =g(s,c ), so any difference between the predicted effects under c and c′c is caused by the context alone. At test time a nuisance shift reverses a training correlation between c and the outcome, a mechanism shift changes physical parameters such as mass, and a composed shift does both; any of these makes a split out-of-distribution (OOD). 3.3 Model and objective Figure 2: Do-JEPA training. Top: the simulator runs the saved state sts_t under do(a)do(a) and do(a∅)do(a_ ); the encoder E (frozen in the slot models shown here) gives the target Δz z. Bottom: the world model encodes the observation into slots ztz_t, routes the action into a slot through the mask m and propagates its effect through the gates GjiG_ji; Δz z is the difference of its two predictions. ℒΔL_ : what the action caused; ℒsupL_sup: where it entered; ℒedgeL_edge: how it travelled; ℒinv,ℒctxL_inv,L_ctx: what must not change. Architecture. The slot predictor for object-aligned variables (Fig. 2) has three parts. (i) A transformer over the history predicts an action-free next state bt+1∈ℝK×db_t+1 ^K× d. (i) An action-entry mask m=softmaxi(hϕ(zti,a)/Tm)⋅[a≠a∅]∈[0,1]K,m=softmax_i (h_φ(z^i_t,a)/T_m )·1[a≠ a_ ]\;∈[0,1]^K, (4) where hϕh_φ scores slot i for action a and TmT_m is a temperature, sums to one for a real action and is zero for a∅a_ . It gates the direct residual ri=miuϕ(zti,a)r^i=m_i\,u_φ(z^i_t,a) of a small network uϕu_φ with bounded (tanh) output. (i) Gated message passing, repeated P times, moves the residual between slots: r←r+γtanh(((A⊙G)rWv⊤)Wo⊤),Gji=σ(qϕ(ztj,zti)/TG),Gjj=0,r← r+γ \! ( ((A G)\,r\,W_v )W_o ), G_ji=σ (q_φ(z^j_t,z^i_t)/T_G ), G_j=0, (5) where r∈ℝK×dr ^K× d has one row per slot, A are attention weights, GjiG_ji gates the message from slot i to slot j and depends on the slot states only, Wv,WoW_v,W_o are bias-free linear maps and γ is a fixed scale. Gated weights are not renormalised, so a closed gate blocks the message. The prediction is z^t+1a=bt+1+r z^\,a_t+1=b_t+1+r. Only r depends on the action and m=0m=0 for a∅a_ , so Δz^t+1=r z_t+1=r. In state-based Push-T the action is known to move the agent, so it enters the agent slot and m is not learned. The LeWM-class backbones (K=1K=1) have no mask or gates. Targets and losses. All targets come from the paired rollouts or the context twin; none uses the simulator’s graph or object identities (Table 6, App. A). The effect loss compares predicted and true paired effects, ℒΔ=1N∑n∥Δz^n−Δzn∥22L_ = 1N _n z_n- z_n _2^2. The support, propagation, invariance and context losses are ℒsup _sup =−1N∑n∑i=1Kysupn,ilogmn,i,ysupn,i=∥Δzn,t+1i∥2∑i′∥Δzn,t+1i′∥2, =- 1N _n _i=1^Ky^sup_n,i m_n,i, y^sup_n,i= z^\,i_n,t+1 ^2 _i z^\,i _n,t+1 ^2, (6) ℒedge _edge =1N∑n(BCEbal(Gn,ynedge)+λsp∥Gn∥1), = 1N _n (BCE_bal (G_n,\,y^edge_n )+ _sp G_n _1 ), (7) ℒinv _inv =1N∑n∑j∉Rn(∥Δz^nj∥22+λg∑iGn,ji), = 1N _n _j∉ R_n ( z^\,j_n _2^2+ _g _iG_n,ji ), (8) ℒctx _ctx =1N∑n∥Δz^n(c)−Δz^n(c′)∥22. = 1N _n \, z_n(c)- z_n(c ) _2^2. (9) ℒsupL_sup is a cross-entropy between the mask and the normalised immediate response energy (an entropy regulariser keeps m sharp); ℒedgeL_edge is a class-balanced binary cross-entropy between gates and onset labels with an ℓ1 _1 penalty; ℒinvL_inv pushes predicted effects on non-responders to zero and closes the gates into them; ℒctxL_ctx asks the predicted effect to be the same under a context and its twin (in state-based Push-T it compares factual predictions of the physical slots; in the end-to-end models it averages Eq. 9 with the same term on the target effects, without stopping gradients). Full objective and model variants. The full objective is ℒ=ℒpred(a)+λrefℒpred(a∅)+λΔℒΔ+λsupℒsup+λedgeℒedge+λinvℒinv+λctxℒctx+ℛ,L=L_pred(a)+ _ref\,L_pred(a_ )+ _ L_ + _supL_sup+ _edgeL_edge+ _invL_inv+ _ctxL_ctx+R, (10) where ℒpred(a)L_pred(a) and ℒpred(a∅)L_pred(a_ ) apply Eq. 2 to the two branches and ℛR collects the backbone’s regulariser and the setting-specific terms of App. D. The variants differ only in how they use the same collected data. Obs uses the factual branch only. Aug, the information-matched control, uses the factual branch, the reference branch and, where used, the context twin as ordinary examples (λref>0 _ref>0, λΔ=0 _ =0) but never compares them. Paired adds ℒΔL_ ; Do-JEPA (full) adds ℒctxL_ctx and, in slot models, the slot losses listed for each setting in Table 8. Mask is our re-implementation of C-JEPA on the same slots. Training, inference and information. Training uses paired trajectories (plus context twins for ℒctxL_ctx). At test time the model receives only an observation history and an action; it predicts an effect by running the predictor with a and with a∅a_ from the same history (Eq. 3), and planning uses only the factual prediction. The losses need an exactly restorable simulator (all losses), object-aligned slots (ℒsupL_sup, ℒedgeL_edge, ℒinvL_inv) and a way to change the context without changing the physics (ℒctxL_ctx). Paired and Aug receive the same branches, so their comparison is matched in information. In the SCMs and state-based Push-T the model’s variables are the physical state; in the pixel settings physical state is used only to build benchmarks and to evaluate (Table 7). 4 Experiments We try to answer four questions, each in the setting that isolates it: Q1: where does the action enter? Q2: where does its effect travel? Q3: what must not change? Q4: does the effect loss help when the representation is learned from pixels? App. C and D describe the settings and training. Protocol. Checkpoints and hyperparameters are selected on in-distribution (IID) validation data, except in two development runs that we mark where they occur. In preregistered experiments, hypotheses, thresholds and seeds were fixed in a protocol file before data collection or before the OOD splits were opened, and OOD splits were evaluated once after the checkpoint was locked. Confirmed means that a result holds on at least 22 of 33 fresh seeds. App. B defines the metrics. 4.1 Q1–Q2: Where does the action enter, and where does its effect travel? Question and setup. Can a model find the directly intervened object, and the paths along which its effect spreads, from paired data alone? Sparsity is often assumed to localise an intervention; if it did, support labels would be unnecessary. A synthetic structural causal model (SCM) gives known ground truth: four objects on a directed ring (i→i+1mod4i→ i+1 4) and three nuisance slots. The action, an impulse at a noisy contact point biased towards the next object, does not name its target; the first step changes only the direct target, and the effect then spreads along the ring for two steps. A hidden context makes the nuisances co-vary with the action in training (ρ=0.95ρ=0.95); the test split reverses this. We compare Mask (global action), Mask+ℒΔ+L_ , a sparse action mask, the mask +ℒΔ+L_ and the mask +ℒΔ+ℒsup+L_ +L_sup, then gates without and with ℒedgeL_edge and the gate term of ℒinvL_inv. In state-based Push-T, where the agent is the known direct target, we ask whether the gate Gblock←agentG_block learns when the agent moves the block. Table 1: Where does the action enter? Synthetic SCM, reversed-correlation test split, seed 7. Top-1 over all seven slots (chance 1/71/7) and, in brackets, over the four objects (chance 0.250.25); nuis. mask: mask mass on nuisance slots; nuis. effect: mean RMS of the predicted effect per nuisance slot. All rows are evaluated with one history slot masked, as in Mask training; mask temperature is 1.01.0 in the sparse-mask rows and 0.70.7 in the last (App. E.1). Best in bold, second best underlined. Top-1 all [obj.] ↑ Target F1F_1 ↑ Nuis. mask ↓ Nuis. effect ↓ Effect MSE ↓ Pred. MSE ↓ Mask (global action) — — — 6.3×10−3¯ 6.3× 10^-3 1.5×10−31.5× 10^-3 3.14×10−3¯ 3.14× 10^-3 Mask +ℒΔ+\,L_ — — — 2.9×−2.9× 10^-3 3.5×−3.5× 10^-4 2.67×−2.67× 10^-3 Sparse mask 0.0000.000 [0.2430.243] 0.0000.000 1.0001.000 2.5×10−22.5× 10^-2 1.1×10−21.1× 10^-2 8.70×10−38.70× 10^-3 Sparse mask +ℒΔ+\,L_ 0.0000.000 [0.2340.234] 0.0000.000 1.0001.000 1.6×10−21.6× 10^-2 1.0×10−21.0× 10^-2 8.55×10−38.55× 10^-3 Sparse mask +ℒΔ+ℒsup+\,L_ +L_sup 0.99950.9995 [0.99950.9995] 0.99980.9998 8.2×−8.2× 10^-5 9.1×10−39.1× 10^-3 9.3×10−4¯ 9.3× 10^-4 3.43×10−33.43× 10^-3 Result: the direct target. A sparse mask selects exactly one slot, as the sparsity term asks, but in every test case it selects a nuisance (top-1 0.0000.000 over all slots; 0.2430.243 if only objects are ranked; Table 1). Adding ℒΔL_ does not help: the propagation step can move the effect wherever it is needed, so any entry slot satisfies the effect loss. Adding ℒsupL_sup, together with a lower mask temperature (we did not run a temperature-matched control), gives 0.99950.9995 with almost no mask mass on nuisances. Sparsity sets how many slots the action enters; here the immediate paired response set which one. Prediction and effect error do not reveal this (Mask+ℒΔ+L_ has the lowest of both), and the support-trained model still predicts effects on nuisances (9.1×10−39.1× 10^-3) because nothing constrains how its effect spreads. An easier SCM shows the same pattern over five seeds (App. E.1). Result: the propagation graph. Gates trained without ℒedgeL_edge are open almost everywhere (edge AUROC 0.6240.624 against the test pairs’ onset labels; Table 12). With ℒedgeL_edge it is 0.9750.975 on all three seeds, and thresholding the mean test gate at 0.50.5 recovers exactly the four ring edges on each seed (mean gate 0.950.95 on ring edges, 0.0020.002–0.0050.005 on other pairs), and predicted nuisance effects fall to 1.9×10−51.9× 10^-5, at the cost of a less accurate direct effect (MSE 0.01370.0137 vs. 0.00330.0033). In state-based Push-T the gate is state-dependent: edge AUROC is 0.9944±0.00020.9944± 0.0002 against 0.5000.500 without gates, and the predicted action effect on nuisances falls from 1.9×10−31.9× 10^-3 (Obs) to 2.1×10−72.1× 10^-7 while prediction MSE differs by at most 2%2\% across models (Table 14). Result: time resolution. Onset labels assume that causal events are separated in time. In a second SCM (six objects on a ring), κ∈1,2,3,4κ∈\1,2,3,4\ of four propagation stages fall inside one observed step (App. E.2). As κ grows, one-hop edge F1F_1 collapses (0.999→0.549→0.502→0.3370.999→ 0.549→ 0.502→ 0.337), while direct-target F1F_1 stays at 0.9970.997 and prediction MSE at 0.01030.0103–0.01050.0105. The onset labels themselves, scored as one-hop edges, fall from F1=1.00F_1=1.00 to 0.250.25: with two transitions between observations, a chain A→B→CA→ B→ C and a fork A→B,A→C\A→ B,A→ C\ produce the same data. Scored against reachability within the observation interval, the same gates reach F1=0.999/0.935/0.992/0.894F_1=0.999/0.935/0.992/0.894, and one gate head per time scale, tied by a reachability constraint, represents both levels (one-hop F1=0.977F_1=0.977; coarse 0.9090.909–0.9770.977). Takeaway. In our synthetic systems, with object-aligned variables and resolvable onsets, the immediate paired response identified the direct target and onset order recovered the ring graph; sparsity and the effect loss did not, and with coarser observation the recoverable relation became reachability, which prediction error does not reveal. On anonymous frozen slots of pixel Push-T, which have no ground-truth graph, a single-seed IID diagnostic gives onset-trained gates that reproduce the test onset labels (AUROC 1.0001.000 vs. 0.8230.823 without ℒedgeL_edge; App. G). 4.2 Q3: What must not change? The action-side losses do not stop a nuisance from changing the prediction. We test whether context twins remove a shortcut through a nuisance that is correlated with the outcome in training. With given variables. In state-based Push-T a nuisance vector is correlated with contact geometry (ρ=0.95ρ=0.95) in training and reversed at test, and the physical inputs are corrupted with sensor noise. Obs degrades by 2.9×2.9× in prediction MSE and the gated model by 3.1×3.1×, although the latter’s effect error changes only by 1.02×1.02×. We chose λctx=0.25 _ctx=0.25 on seed 7’s IID–OOD trade-off curve, so that seed’s OOD split informed the choice. On the two other seeds, ℒctxL_ctx lowers OOD error by 43.9%43.9\% and 55.2%55.2\% and closes 71.9%71.9\% and 85.4%85.4\% of the IID-to-OOD gap, at 4.2%4.2\% and 3.5%3.5\% higher IID error, with edge AUROC unchanged (App. E.3). Boundary condition: learned pixel latents. On learned latents the same idea needs care. Applied as a generic, isotropic penalty on latent effects, ℒctxL_ctx lowers the latent context shift of predicted effects (20%20\% on frozen VideoSAUR slots, 4646–47%47\% end to end in CausalWorld) but costs factual and effect accuracy, and on frozen slots the shift decoded to pixels does not fall. Diagnostics trace this to a mismatch between latent and physical distance: two latent directions carry about 99.8%99.8\% of the decoded physical error, and a predictor-side variant of the penalty leaves 1010–19%19\% more residual energy exactly there (App. E.6). Pairing alone already lowers the latent context shift by about 66%66\% without any context term (Sec. 4.3). Takeaway. With given variables, nuisance interventions remove most of the shortcut at a small IID cost. On learned latents, invariance has to be measured in the directions that carry the physics; a generic latent penalty is not enough. 4.3 Q4: Does the effect loss help from pixels? On C-JEPA’s own frozen VideoSAUR slots of pixel Push-T, Do-JEPA beats C-JEPA-style masking on all six metrics and all three seeds: factual error is 30%30\% lower, responsive effect error 12%12\% lower, effect cosine 0.140.14 higher, response AUROC 0.210.21 higher (masking is at chance) and the context shift of predicted effects 31%31\% lower in pixels (App. E.4). We test the effect loss itself where the representation is learned, end to end, against Aug, which sees exactly the same branches; this control matters because Paired receives both branches. LeWM on pixel Push-T. We fine-tune the released LeWM checkpoint (Maes et al., 2026b) for 1010 epochs on 10,00010,000 exact same-state pairs (a 55-step action macro vs. a zero macro; half responsive), with and without ℒΔL_ (weight 0.50.5). The preregistered rule needs 22 of 33 fresh seeds to lower effect RMSE by 5%5\% or raise effect cosine by 0.050.05. All three pass (Fig. 3): effect RMSE falls by 28.4%28.4\% (0.213→0.1530.213→ 0.153; per seed 27.527.5–29.0%29.0\%), cosine over all pairs rises by 0.0660.066, and factual MSE also falls, by 39.7%39.7\%. All are latent metrics, each in its own model’s latent space. On a fresh holdout of 5,0005,000 pairs with actions drawn independently from the expert action distribution and no outcome filtering (6.5%6.5\% responsive), effect RMSE is 11.2%11.2\% lower (3/33/3 seeds) and factual MSE 17.5%17.5\% lower, mostly on non-responsive pairs. A physical read-out, a linear map from latent effects to block displacement fitted per model, confirms the gain on the training distribution (responsive physical effect error 7%7\% lower on every seed) but not on the holdout’s responsive pairs (3%3\% higher; App. E.5). The gain follows the training intervention distribution. Trained with the same recipe on natural expert action sequences, Paired lowers the responsive physical effect error by 13.5%13.5\% on an episode-disjoint natural test set and by 5.5%5.5\% on the outcome-unfiltered holdout, on every seed. Figure 3: Effect loss on LeWM, pixel Push-T. Aug and Paired see the same paired images; Paired adds ℒΔL_ . Latent metrics; means over three fresh seeds (dots: seeds); red: relative change (absolute for the cosine, taken over all pairs). Panels 1–3: held-out pairs from the training distribution (half responsive); panel 4: outcome-unfiltered holdout (6.5%6.5\% responsive). CausalWorld: three independently generated benchmarks. In the CausalWorld pushing task (Ahmed et al., 2020) a robot finger pushes a block; the branches differ only in the first 9-D joint command, which the reference mirrors about the current joint position. OOD splits reverse a visual shortcut (visual), change mass and friction while making the context independent of the outcome (mechanism), or change the physics and reverse the shortcut (composed). A first version of the benchmark leaked the label through how samples were drawn (sampling-only AUROC 0.8340.834). We rebuilt it with outcome-defined labels and exact covariate balance (AUROC 0.500.50; App. C.5), regenerated it from three dataset seeds, and fixed in advance the expected signature of Paired against Aug: lower responsive effect error and context shift and higher cosine (benefit), higher factual error and lower response AUROC (cost). Checkpoints were locked on IID data before the OOD splits were opened once. Table 2: CausalWorld: Paired vs. Aug on three independently generated benchmarks (dataset seeds 17/27/37, model seed 107, 4040k updates), locked OOD splits. Mean relative changes; brackets count instances that move in the stated direction; “Signature” counts instances with the full preregistered benefit and cost. ∗The mechanism split also decorrelates the visual context (App. C.5). Benefit Cost OOD split Resp. effect RMSE Effect cosine Latent ctx. shift Factual RMSE Response AUROC Signature Visual 3.9%3.9\% lower [2/3] +0.018+0.018 [3/3] 66.2%66.2\% lower [3/3] 52.8%52.8\% higher [3/3] −0.054-0.054 [3/3] 2/32/3 Mechanism∗ 20.3%20.3\% lower [3/3] +0.030+0.030 [2/3] 65.4%65.4\% lower [3/3] 51.5%51.5\% higher [3/3] −0.070-0.070 [3/3] 2/32/3 Composed 19.5%19.5\% lower [3/3] +0.020+0.020 [2/3] 66.2%66.2\% lower [3/3] 51.1%51.1\% higher [3/3] −0.073-0.073 [3/3] 2/32/3 Result. The preregistered signature holds on all three OOD splits (Table 2). Paired lowers the context shift of predicted effects by about 66%66\% without any context loss, and lowers responsive effect error by about 20%20\% under the mechanism and composed shifts, at the cost of 5151–53%53\% higher factual error and lower response AUROC. Three fresh model seeds on one benchmark instance show the same benefit and cost. In these runs the factual cost (1515–18%18\%) exceeds the preregistered 10%10\% tolerance. Where the factual cost comes from. In Push-T the effect loss fine-tunes a pretrained model and in CausalWorld it shapes the representation from scratch. To separate the two, we fine-tune each converged Aug model for 2,0002,000 updates with and without ℒΔL_ (the Push-T regime) and compare the last checkpoints. The factual cost disappears: factual error changes by −4-4 to +2%+2\% and response AUROC is unchanged on every instance, while responsive effect error falls by 8%8\% IID and 13%13\% under the physics shifts and the effect cosine rises on every instance and split (Table 3). From scratch, the effect weight sets the trade-off: α=0.25α=0.25 cuts the factual cost from 52%52\% to 10%10\% and the AUROC loss from 0.0660.066 to 0.0160.016, and keeps the IID effect gain and about two thirds of the gain under physics shifts. Table 3: When does the effect loss cost factual accuracy? CausalWorld, Paired against the Aug model of the same regime, mean relative change over the three benchmark instances. OOD: mean over the three OOD splits; physics: mechanism and composed splits. Last column: instances whose factual error rises by at most 10%10\% on every split. The last two rows are post-hoc diagnostics (App. E.6). Factual RMSE Resp. effect RMSE Response Latent ctx. Factual Regime IID OOD IID Physics AUROC shift within 10%10\% From scratch, α=1α=1 +51.9%+51.9\% +51.8%+51.8\% −9.0%-9.0\% −19.9%-19.9\% −0.066-0.066 −66.0%-66.0\% 0/30/3 From scratch, α=0.25α=0.25 +10.3%+10.3\% +10.6%+10.6\% −9.1%-9.1\% −12.8%-12.8\% −0.016-0.016 −33.6%-33.6\% 1/31/3 Fine-tuned from Aug, α=1α=1 −0.6%-0.6\% −0.9%-0.9\% −8.2%-8.2\% −13.3%-13.3\% +0.008+0.008 −26.2%-26.2\% 3/33/3 Takeaway. In every comparison on end-to-end LeWM-class models, the effect loss lowered mean effect error relative to an information-matched control. Its effect on factual accuracy depends on the training regime: fine-tuning an existing model with it keeps factual accuracy (CausalWorld) or improves it (Push-T), while training from scratch trades factual accuracy and response AUROC for a larger effect gain, in proportion to the effect weight. 5 Discussion and Limitations We show direct-target detection and responder prediction with object-aligned variables (Q1–Q2), nuisance robustness with given variables (Q3), and better effect prediction from pixels than an information-matched control, including lower responsive effect error under CausalWorld physics shifts (Q4). Paired supervision can also be added to a pretrained LeWM planner through a centered adapter, with success equivalent to the base planner within ±3± 3 percentage points (App. F). Our supervision signals capture how interventions propagate in observed rollouts, under the training intervention distribution and at the observation rate. Learning them does not require identification of the underlying mechanism, and we do not claim it. Here do(⋅)do(·) names interventions and we use no do-calculus. Paired effects are counterfactual in the sense that both branches share state and noise in a simulator. Limitations. Do-JEPA rests on several assumptions. First, exact pairs require a simulator whose full state can be saved and restored, and each pair costs two rollouts (a context twin changes only the observation, not the rollout). Real trials share neither noise nor exact state, so differences between them are noisy and may be biased. Second, the intervention and its reference must be known, although the direct target need not be. Third, support and onset labels require object-aligned slots, a known slot count, a single direct target per action (the support head is a softmax) and a frame rate at which response onsets separate in time. Finally, context twins assume that the context can change without changing the physics, and the nuisances in our benchmarks are rendered or synthetic. Relaxing these assumptions, for example with approximate real-world pairs or anonymous object slots, is a promising direction for future work. 6 Conclusion In this work, we proposed Do-JEPA, a framework for training latent world models with interventions on the world rather than on what the model sees. From one saved simulator state, Do-JEPA contrasts an action with a reference action and learns what the action changed, where it entered, where its effect travelled and what must stay invariant. With object-aligned variables in synthetic systems, the immediate paired response identifies where an action enters, and onset order identifies where its effect travels, down to the time resolution at which events separate. From pixels, the paired effect loss improves effect prediction over a control trained on the same data in two environments, and fine-tuning with it preserved factual accuracy in our experiments. Context twins provide nuisance invariance with given variables, whereas on learned latents invariance must be measured in the directions that carry the physics. Finally, prediction error revealed little about these properties, which suggests that interventional metrics should become a routine part of world-model evaluation. References Ahmed et al. (2020) O. Ahmed, F. Träuble, A. Goyal, A. Neitz, Y. Bengio, B. Schölkopf, M. Wüthrich, and S. Bauer CausalWorld: a robotic manipulation benchmark for causal structure and transfer learning. arXiv preprint arXiv:2010.04296. Cited by: §C.5, §4.3. Ahuja et al. (2023) K. Ahuja, D. Mahajan, Y. Wang, and Y. Bengio Interventional causal representation learning. In International Conference on Machine Learning (ICML), Cited by: §2. Assran et al. (2025) M. Assran A. Bardes et al. V-jepa 2: self-supervised video models enable understanding, prediction and planning. arXiv preprint arXiv:2506.09985. Cited by: §1, §2. Assran et al. (2023) M. Assran, Q. Duval, I. Misra, P. Bojanowski, P. Vincent, M. Rabbat, Y. LeCun, and N. Ballas Self-supervised learning from images with a joint-embedding predictive architecture. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §1, §2. Balestriero and LeCun (2025) R. Balestriero and Y. LeCun LeJEPA: provable and scalable self-supervised learning without the heuristics. arXiv preprint arXiv:2511.08544. Cited by: §1, §2. Brehmer et al. (2022) J. Brehmer, P. De Haan, P. Lippe, and T. Cohen Weakly supervised causal representation learning. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §2. Chi et al. (2023) C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song Diffusion policy: visuomotor policy learning via action diffusion. In Robotics: Science and Systems (RSS), Cited by: §C.2. French (1999) R. M. French Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences 3 (4), p. 128–135. Cited by: Appendix F. García et al. (1989) C. E. García, D. M. Prett, and M. Morari Model predictive control: theory and practice — a survey. Automatica 25 (3), p. 335–348. Cited by: §2. Ha and Schmidhuber (2018) D. Ha and J. Schmidhuber Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §1. Hafner et al. (2023) D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104. Cited by: §1. Houlsby et al. (2019) N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. de Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning (ICML), Cited by: Appendix F. Hu et al. (2022) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), Cited by: Appendix F. Kirkpatrick et al. (2017) J. Kirkpatrick, R. Pascanu, N. Rabinowitz, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences 114 (13), p. 3521–3526. Cited by: Appendix F. LeCun (2022) Y. LeCun A path towards autonomous machine intelligence. Technical report OpenReview. Note: Version 0.9.2 Cited by: §1, §2. Lei et al. (2022) A. Lei, B. Schölkopf, and I. Posner Variational causal dynamics: discovering modular world models from interventions. arXiv preprint arXiv:2206.11131. Cited by: §2. Li et al. (2020) Y. Li, A. Torralba, A. Anandkumar, D. Fox, and A. Garg Causal discovery in physical systems from videos. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §2. Lippe et al. (2022) P. Lippe, S. Magliacane, S. Löwe, Y. M. Asano, T. Cohen, and E. Gavves CITRIS: causal identifiability from temporal intervened sequences. In International Conference on Machine Learning (ICML), Cited by: §2. Lippe et al. (2023) P. Lippe, S. Magliacane, S. Löwe, Y. M. Asano, T. Cohen, and E. Gavves BISCUIT: causal representation learning from binary interactions. In Conference on Causal Learning and Reasoning (CLeaR), Cited by: §2. Liu et al. (2023) Y. Liu, A. Alahi, C. Russell, M. Horn, D. Zietlow, B. Schölkopf, and F. Locatello CausalTriplet: an open challenge for intervention-centric causal representation learning. In Conference on Causal Learning and Reasoning (CLeaR), Cited by: §2. Maes et al. (2026a) L. Maes, Q. Le Lidec, L. Facury, N. Massaudi, A. Chaurasia, F. Capuano, R. Gao, T. Gillin, D. Haramati, D. Scieur, Y. LeCun, and R. Balestriero stable-worldmodel: a platform for reproducible world modeling research and evaluation. arXiv preprint arXiv:2605.21800. Cited by: §C.4, Appendix F. Maes et al. (2026b) L. Maes, Q. Le Lidec, D. Scieur, Y. LeCun, and R. Balestriero LeWorldModel: stable end-to-end joint-embedding predictive architecture from pixels. arXiv preprint arXiv:2603.19312. Cited by: Appendix F, §2, §4.3. Nam et al. (2026) H. Nam, Q. Le Lidec, L. Maes, Y. LeCun, and R. Balestriero Causal-JEPA: learning world models through object-level latent masking. arXiv preprint arXiv:2602.11389. Cited by: Figure 1, §1, §2. Oquab et al. (2024) M. Oquab, T. Darcet, T. Moutakanni, et al. DINOv2: learning robust visual features without supervision. Transactions on Machine Learning Research. Cited by: §E.4. Rubinstein (1999) R. Y. Rubinstein The cross-entropy method for combinatorial and continuous optimization. Methodology and Computing in Applied Probability 1 (2), p. 127–190. Cited by: §2. Schölkopf et al. (2021) B. Schölkopf, F. Locatello, S. Bauer, N. R. Ke, N. Kalchbrenner, A. Goyal, and Y. Bengio Toward causal representation learning. Proceedings of the IEEE 109 (5), p. 612–634. Cited by: §2. Sontakke et al. (2021) S. A. Sontakke, A. Mehrjou, L. Itti, and B. Schölkopf Causal curiosity: RL agents discovering self-supervised experiments for causal representation learning. In International Conference on Machine Learning (ICML), Cited by: §2. von Kügelgen et al. (2021) J. von Kügelgen, Y. Sharma, L. Gresele, W. Brendel, B. Schölkopf, M. Besserve, and F. Locatello Self-supervised learning with data augmentations provably isolates content from style. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §2. Zadaianchuk et al. (2023) A. Zadaianchuk, M. Seitzer, and G. Martius Object-centric learning for real-world videos by predicting temporal feature similarities. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §E.4. Zhou et al. (2024) G. Zhou, H. Pan, Y. LeCun, and L. Pinto DINO-WM: world models on pre-trained visual features enable zero-shot planning. arXiv preprint arXiv:2411.04983. Cited by: §1, §2. Contents of the Appendix A Notation and Terminology.A B Metrics Guide.B C Benchmarks.C C.1 Synthetic SCMs.C.1 C.2 State-based Push-T.C.2 C.3 Vision Push-T.C.3 C.4 LeWM Push-T pairs and planning corpora.C.4 C.5 CausalWorld pushing.C.5 D Training and Evaluation Details.D E Additional Results.E E.1 Synthetic SCMs.E.1 E.2 Temporal resolvability.E.2 E.3 State-based Push-T.E.3 E.4 Vision Push-T.E.4 E.5 LeWM Push-T: per-seed results.E.5 E.6 CausalWorld.E.6 F Planning Study Details.F G Boundary Conditions and Diagnostic Findings.G Table 4: Main results at a glance. Each row gives the strongest evidence for one finding and where it is reported. Seeds are model seeds; “instances” are independently generated benchmarks. Finding Evidence Where Support supervision finds where the action enters direct target found in 99.95%99.95\% of test cases; a sparse mask picks a nuisance slot in every case Table 1 Onset supervision recovers where the effect travels edge AUROC 0.9750.975 vs. 0.6240.624; the four ring edges recovered exactly on 3/33/3 seeds Table 12 Gates confine effects to the right objects state Push-T edge AUROC 0.9940.994 vs. 0.5000.500; predicted effect on nuisances 1.9×10−3→2.1×10−71.9× 10^-3→ 2.1× 10^-7 Table 14 Context twins remove a nuisance shortcut on the two seeds not used to pick the weight: OOD error 4444–55%55\% lower, 7272–85%85\% of the IID-to-OOD gap closed, IID cost ≤4.2%≤ 4.2\% App. E.3 Intervention beats masking on C-JEPA’s own slots all six metrics, 3/33/3 seeds: factual error 30%30\% lower, AUROC +0.21+0.21, pixel context shift 31%31\% lower Table 15 Effect loss helps from pixels (LeWM) latent effect error 28.4%28.4\% lower (3/33/3); physical effect error 13.5%13.5\% lower when trained and tested on natural sequences (3/33/3) Fig. 3, Table 18 Effect loss helps under physics shifts (CausalWorld) responsive effect error ≈20%≈20\% lower and latent context shift 66%66\% lower, on three benchmark instances Table 2 Fine-tuning keeps factual accuracy factual error −4-4 to +2%+2\%, effect error 88–13%13\% lower, on 3/33/3 instances Tables 3, 21 Compatible with a pretrained planner success within ±3± 3 points of the base planner (600600 episodes) App. F Appendix A Notation and Terminology Table 5 lists every symbol used in the paper. The column “Role” says whether a quantity is observed (produced by the simulator or the encoder), predicted (produced by the predictor), a target (used inside a loss), a label (a target derived from paired rollouts), or a parameter. The column “When” says whether it exists at training time, at test time, or only for evaluation. Table 5: Notation. K: number of slots; d: slot dimension; H: history length; k: rollout horizon; N: batch size. Symbol Meaning Shape Role When sts_t physical simulator state env.-specific observed build/eval a,a∅a,\ a_ action; reference action (a physical no-op unless stated) A input train, test c,c′c,\ c context (nuisance) value and its twin env.-specific input train (twin), test (c) u confounder: affects action choice and outcome scalar hidden build εt _t exogenous simulator noise, shared by both branches — hidden build oto_t observation g(st,c)g(s_t,c) (image or state vector) env.-specific observed train, test zt=(zt1,…,ztK)z_t=(z^1_t,…,z^K_t) encoder output, one row per slot K×dK× d observed train, test zt+ℓa,zt+ℓa∅z^a_t+ ,\ z^a_ _t+ encodings of the factual and reference branches K×dK× d target train z^t+ℓa,z^t+ℓa∅ z^\,a_t+ ,\ z^\,a_ _t+ predictions for the two branches K×dK× d predicted train, test Δz z true paired effect, Eq. 3 K×dK× d target train Δz z predicted paired effect, Eq. 3 K×dK× d predicted train, test Δp,Δp p,\ p true and decoded physical effect of the manipulated object 22 or 33 observed / predicted eval m action-entry mask, Eq. 4 K predicted train, test r direct (then propagated) residual K×dK× d predicted train, test GjiG_ji gate on the message from slot i to slot j, Eq. 5 K×K× K predicted train, test A attention weights between slots K×K× K predicted train, test ysupy^sup normalised immediate response energy (support label) K label train yjiedgey^edge_ji onset-order propagation label K×K× K label train R response set: slots that respond within k steps set label train T true direct target set observed eval only Eψ,FθE_ψ,\ F_θ encoder and predictor — parameter — hϕ,uϕ,qϕh_φ,\ u_φ,\ q_φ mask scorer, direct-effect network, gate scorer — parameter — Fbase,Fadapt,gϕF_base,\ F_adapt,\ g_φ frozen planner predictor, adapted predictor, adapter — parameter — κ number of causal stages hidden in one observed step integer design build λ(⋅),β _(·),\ β loss weights (β = context weight in CausalWorld) scalar hyperparameter train Terminology. We use each term in one sense only. Intervention, do(a)do(a). Setting the action to a chosen value in a simulator from a saved state. We use the notation to name interventions, not to invoke do-calculus. Reference action a∅a_ . The action of the second branch. It is a physical no-op in Push-T and the synthetic SCMs, a mirrored joint command in CausalWorld, and a different real expert action in the reference-contrast test (App. F). Paired trajectory. Two rollouts from one saved state with shared context and noise that differ only in the intervention. Counterfactual refers only to this construction. Direct intervention, intervention target, direct target. The variables an intervention changes at once. Intervention support (direct support) is its estimate from the first step of a pair. Response, responsive component. A variable responds if its paired effect exceeds a threshold within the horizon; the response set contains all such variables. A responsive pair is a pair in which the manipulated object moves by more than a threshold: 22 px in vision Push-T, 22 px or 0.020.02 rad in the LeWM Push-T corpora, and 22 m in CausalWorld, where pairs between 0.50.5 and 22 m are discarded (App. C). Downstream (propagated) effect. A response of a variable that is not a direct target. Temporal support, onset. The step at which a variable starts to respond. Causal propagation is the set of ordered pairs (i,j)(i,j) such that j starts to respond one step after i. Invariant variable, invariance. A variable outside the response set of a pair; the model should predict zero effect on it. Context, nuisance context. A variable that changes the observation but not the physics. Context invariance means that predicted effects do not change when only the context changes. Mechanism, mechanism shift. The physical parameters of the dynamics (mass, friction, moment of inertia); a mechanism shift changes them at test time. A composed shift changes both mechanism and context. OOD. A test split with a nuisance, mechanism, composed or confounder shift; IID means the training distribution. Targets and information flow. Table 6 lists each loss with its target and the source of that target, and Table 7 lists where each kind of information enters training, testing and evaluation. Table 6: Losses and their targets. Every target is computed from paired rollouts or context twins at training time. ei=∥Δzt+1i∥2e_i= z^\,i_t+1 ^2 is the immediate response energy of slot i; R is the response set; “onset order” is defined in Sec. 3.2. Loss Question Target Source of the target ℒpredL_pred what happens next? zt+1az^a_t+1, zt+1a∅z^a_ _t+1 observed next frames ℒΔL_ what did the action change? Δz=za−za∅ z=z^a-z^a_ paired rollouts, Eq. 3 ℒsupL_sup where did it enter? yisup=ei/∑i′ei′y^sup_i=e_i/ _i e_i first step of the pair ℒedgeL_edge where did it travel? yjiedge∈0,1y^edge_ji∈\0,1\ onset order in the pair ℒinvL_inv what must not change? Δz^j=0 z^\,j=0 for j∉Rj∉ R response set of the pair ℒctxL_ctx what must ignore context? Δz^(c)=Δz^(c′) z(c)= z(c ) context twin Table 7: Where information enters. “Input” means the model receives it; “target” means a loss uses it; “build/eval” means it is used only to construct benchmarks or to compute metrics. Test-time model inputs are the observation history and the action only. Information Training Test Build/eval only Observation history, action a input input — Reference branch oa∅o^a_ input (Aug), target (ℒΔL_ ) — — Context twin o′o input (Aug), target (ℒctxL_ctx) — — Support, onset and response labels target (from the pair) — — Object count and slot alignment architecture (slot models) architecture synthetic metrics Ground-truth graph, object identities — — metrics Physical state, SCMs and state Push-T input, target input metrics Physical state, pixel settings (positions, mass, friction) — — balancing, probes Appendix B Metrics Guide For each metric we give its purpose, inputs, definition, range, better direction, interpretation, where it is used, and caveats. n indexes test examples; P is the set of responsive pairs; N is the set of nuisance slots. Physical quantities (px in Push-T, m in CausalWorld) are decoded from latents by a read-out probe trained on training and IID data only. Relative change and relative reduction. For a metric M where lower is better, the relative reduction of a method over a baseline is relative reduction=100⋅Mbase−MmethodMbase%,relative reduction=100· M_base-M_methodM_base\ \%, so a positive value means the method is better. The relative change is 100(Mmethod−Mbase)/Mbase100\,(M_method-M_base)/M_base, the negative of the relative reduction. In the text we write “X%X\% lower” or “X%X\% higher”. Rates (success, accuracy) are compared in percentage points (p): a change from 83.0%83.0\% to 82.3%82.3\% is −0.7p-0.7\,p, which is a relative change of −0.8%-0.8\%. Cosine and AUROC differences are absolute differences. Factual prediction error (↓ ). Purpose: how well the model predicts what actually happens. Inputs: predicted and target latents of the factual branch; for the physical version, a probe D from latents to physical state s. Definition: MSElat=1N∑n∥z^t+1,n−zt+1,n∥2MSE_lat= 1N _n z_t+1,n-z_t+1,n ^2; RMSEphys=(1N∑n∥D(z^t+1,n)−st+1,n∥2)1/2RMSE_phys= ( 1N _n D( z_t+1,n)-s_t+1,n ^2 )^1/2. Range: [0,∞)[0,∞). Interpretation: the “cost” side of the trade-offs in Secs. 4.2–4.3. Used in: all settings (latent MSE in LeWM Push-T; block-position RMSE in vision Push-T and CausalWorld). Caveats: latent and physical versions can disagree: a model can have lower latent MSE and higher physical error (App. E.6). Good prediction does not imply correct causal structure (Sec. 4.1). Effect error: effect MSE and effect RMSE (↓ ). Purpose: how accurately the model predicts the difference an action makes. Inputs: Δz z and Δz z (Eq. 3) for the same state. Definition: RMSEΔ=(1N∑n∥Δz^n−Δzn∥2)1/2RMSE_ = ( 1N _n z_n- z_n ^2 )^1/2; effect MSE is its square. Direct-effect MSE restricts the sum to the directly intervened slot. In code, latent MSEs (effect and prediction) also average over the KdKd coordinates: they equal the square of the formula above divided by KdKd. Range: [0,∞)[0,∞). Interpretation: the quantity ℒΔL_ optimises. Used in: all settings; the primary metric of the LeWM and planning studies. Caveats: if few pairs are responsive, the average is dominated by null effects (Sec. 4.3). A model with the wrong mechanism can have the lowest effect error (Table 1). Responsive effect RMSE (↓ ). Purpose: effect accuracy only where something happened. Inputs: decoded physical effects Δp^n,Δpn p_n, p_n of the manipulated object on responsive pairs. Definition: RMSE=(1||∑n∈∥Δp^n−Δpn∥2)1/2RMSE_P= ( 1|P| _n p_n- p_n ^2 )^1/2, =n:∥Δpn∥>2P=\n: p_n >2\ (px or m; App. C gives the exact rule per benchmark). Range: [0,∞)[0,∞), in px or m. Interpretation: the main “benefit” metric in vision Push-T and CausalWorld. Caveats: depends on the threshold and on probe accuracy; with few responsive pairs it rests on a few hundred examples. Effect cosine (↑ ). Purpose: whether the predicted effect points the right way, ignoring size. Inputs: predicted and true effect vectors on responsive pairs (physical, or latent where stated). Definition: cos¯=1||∑n∈⟨Δp^n,Δpn⟩/(∥Δp^n∥∥Δpn∥) = 1|P| _n p_n, p_n /( p_n \, p_n ). Range: [−1,1][-1,1]; 00 means unrelated directions. Interpretation: the “how” of an intervention. Used in: vision Push-T, LeWM Push-T (latent; over all pairs unless the responsive subset is named), CausalWorld. Caveats: noisy for near-zero effects and blind to magnitude; latent cosine can stay high while physical cosine drops. Response AUROC (↑ ). Purpose: whether the model knows if an action will reach the object. Inputs: a score per pair, σn=∥Δp^n∥ _n= p_n , and the physical responsive/non-responsive label. Definition: AUROC=Pr(σi>σj∣i responsive,j non-responsive)AUROC= ( _i> _j i responsive,\,j non-responsive). Range: [0,1][0,1]; 0.50.5 is chance; below 0.50.5 means reversed scores. Interpretation: detection of responses, not their size or direction. Used in: vision Push-T, CausalWorld, LeWM (App. G). Caveats: can be inflated by how samples were drawn (sampling-only AUROC 0.830.83 before the CausalWorld rebuild). Detecting a response is not predicting it. Direct-target accuracy and F1F_1 (↑ ). Purpose: whether the action is routed into the right slot. Inputs: mask m; true direct target T (evaluation only). Definition: top-1 accuracy 1N∑n[argmaxi∈mn,i∈Tn] 1N _n1[ _i m_n,i∈ T_n] for a candidate set C: all K slots (all-slot) or the object slots only (object-only, the value our evaluators log); F1F_1 of [mi≥1/K]1[m_i≥ 1/K] against [i∈T]1[i∈ T]. Range: [0,1][0,1]; chance top-1 is 1/K1/K for all slots (1/71/7 in the hard SCM) and 1/Kobj1/K_obj for objects only (0.250.25). Used in: synthetic SCMs. Caveats: object-only top-1 can look like chance when the mask puts almost all its mass on nuisances (Table 1: 0.2430.243 object-only, 0.0000.000 all-slot); read it with the nuisance mask. It saturates on easy benchmarks (every variant reaches object-only top-1 of 1.01.0 in the easy SCM) and needs object-aligned slots. Mask leakage: nuisance mask (↓ ). Purpose: how much of the action enters slots it should not enter. Definition: m¯=1N∑n∑i∈mn,i m_N= 1N _n _i m_n,i. Range: [0,1][0,1]. Used in: synthetic SCMs. Caveats: a clean mask does not imply clean effects: the support-trained model has nuisance mask 8.2×10−58.2× 10^-5 but still predicts nuisance effects. Nuisance effect (↓ ). Purpose: how much the model believes the action moves variables it cannot move. Definition: the mean per-slot RMS of the predicted effect on nuisance slots, E=1N∑n1||∑j∈(1d∥Δz^nj∥2)1/2E_N= 1N _n 1|N| _j ( 1d z^\,j_n ^2 )^1/2. Range: [0,∞)[0,∞), in latent units (not squared). Used in: synthetic SCMs, state Push-T. Caveats: scale-dependent; zero for a model that predicts no effects at all, so always read together with effect error. Edge AUROC and gate means (↑ AUROC, ↑ true gate, ↓ false gate). Purpose: whether gates rank the edges an effect actually used above the other slot pairs. Inputs: gates GjiG_ji of each test example; its onset labels yjiedgey^edge_ji, computed from the test pair’s rollouts as in training. Definition: AUROC of Gji\G_ji\ against yjiedge\y^edge_ji\ over all off-diagonal slot pairs of all test examples (event-level); mean gate over positive (“true”) and negative (“off-path”) pairs. Recovery of the structural graph is scored separately, by thresholding the mean test gate (structural F1F_1 below). Range: [0,1][0,1]; AUROC 0.50.5 is uninformative. Used in: synthetic SCMs, state Push-T, pixel Push-T diagnostic (App. G). Caveats: a correctly learned edge that a rollout did not use counts as a negative; the gates see slot states, not the action, so they cannot represent labels that differ between actions in the same state. In state Push-T the event-level edge precision equals the contact rate (0.7240.724) and is not reported. Structural, event-level and schedule-aware F1F_1 (↑ ). Purpose: agreement of the thresholded graph with a reference relation. Inputs: G^=[G>0.5] G=1[G>0.5] and a reference: the one-hop structural graph, the edges active in a rollout, or reachability within the observation interval of the schedule. Definition: F1(G^,reference)F_1( G,\,reference) over off-diagonal slot pairs. Range: [0,1][0,1]. Used in: synthetic SCMs (Sec. 4.1, App. E.2). Caveats: the wrong reference gives misleading numbers: event-level F1F_1 is 0.6670.667 where structural F1F_1 is 1.01.0. Temporal resolvability (↑ ). Purpose: whether the order of causal events is visible in the data. Inputs: per-slot effect curves ei(t)=∥Δzti∥e_i(t)= z^\,i_t from paired rollouts; slots matched over time. Definition: onset ti=mint:ei(t)≥τt_i= \t:e_i(t)≥τ\; margin between the first and second onsets; usable fraction of pairs with a unique earliest responder; temporal-label F1F_1 of the onset labels against one-hop edges. Range: fractions and F1F_1 in [0,1][0,1]; margin in steps. Used in: temporal SCM; pixel preflight (App. G). Caveats: equal onsets are unresolved; results depend on τ. Context shift of predicted effects (↓ ). Purpose: how much a change that should not matter alters the predicted effect. Inputs: predicted effects for the same state and action under c and c′c . Definition: latent: Sctx=1N∑n∥Δz^n(c)−Δz^n(c′)∥S_ctx= 1N _n z_n(c)- z_n(c ) ; physical: the RMSE between the two decoded block effects, in px or m. Range: [0,∞)[0,∞). Used in: vision Push-T, LeWM (App. G), CausalWorld. Caveats: latent and physical versions can disagree (Table 15); the true latent effect can itself depend on the context. OOD/IID ratio and gap reduction (↓ ratio, ↑ gap reduction). Purpose: how much a correlation reversal hurts, and how much a method helps. Definition: ρ=εOOD/εIIDρ= _OOD/ _IID; gap reduction =1−(εOODnew−εIIDnew)/(εOODold−εIIDold)=1-( ^new_OOD- ^new_IID)/( ^old_OOD- ^old_IID). Range: ρ≥0ρ≥ 0 (11 = no degradation). Used in: state Push-T. Caveats: ρ also falls if IID error rises, so report both errors. Physically weighted error (↓ ). Purpose: error measured in the latent directions that change physics. Inputs: a linear physical decoder WDW_D (latent to physics) fitted with privileged labels. Definition: EW=∥WD(Δz^−Δz)∥2E_W= W_D( z- z) ^2 versus Eraw=∥Δz^−Δz∥2E_raw= z- z ^2. Used in: CausalWorld diagnosis (App. E.6). Caveats: a diagnostic only; it needs privileged labels. Standardised mean difference, SMD (|⋅|↓|·| ). Purpose: whether responsive and non-responsive pairs differ before the intervention. Definition: (x¯resp−x¯non)/(sresp2+snon2)/2( x_resp- x_non)/ (s^2_resp+s^2_non)/2 for a pre-intervention covariate x. Range: real; |SMD|≤0.10|SMD|≤ 0.10 counts as balanced. Used in: CausalWorld construction (App. C.5). Caveats: checks means of measured covariates only. Sampling-only AUROC (→0.5→ 0.5). Purpose: whether the label can be guessed from how a sample was drawn. Inputs: only the variables used to construct the sample (initial geometry, action, warm-up length), never its outcome. Definition: AUROC of a classifier trained on these variables to predict the response label. Range: [0,1][0,1]; the ideal is 0.50.5; our threshold was 0.600.60. Used in: CausalWorld construction. Caveats: detects shortcuts only through the variables it is given. Planning success (↑ ). Purpose: whether the model is useful for control. Definition: 1E∑e[goal reached within the budget] 1E _e1[goal reached within the budget] over E fixed start–goal episodes, with the official success criterion. Range: [0,100]%[0,100]\%. Used in: App. F. Caveats: dominated by the planner and its cost; with 5050 episodes, differences of a few points are not significant. Different episode sets give different reference values (94%94\%, 83.0%83.0\%, 87.8%87.8\%). Rollout ratio (→1→ 1; eligible if ≤1.2≤ 1.2). Purpose: how much adaptation changed the base dynamics. Definition: r=εadapted/εbaser= _adapted/ _base, the 55-step open-loop latent error on expert trajectories. Used in: planning adapters. Caveats: one-step versions are weak proxies: one-step cosine 0.9990.999 with the base model still lost 16p16\,p of planning. Appendix C Benchmarks Table 8 summarises the settings; the subsections below describe each benchmark and Fig. 4 shows its nuisances. Table 8: Settings. Terms are the losses of Eq. 10 that are switched on. Seeds are model seeds unless stated. Setting Model input Intervention vs. reference Shift at test Terms used Train/val/test Seeds Synthetic SCM 77 slots, d=16d=16 impulse at a noisy contact point vs. none reversed action–nuisance corr. ℒΔ,ℒsup,ℒedge,ℒinvL_ ,L_sup,L_edge,L_inv 55k/11k/22k 11–33 Temporal SCM 99 slots, d=18d=18 impulse vs. none observation stride κ ℒΔ,ℒsup,ℒedge,ℒinvL_ ,L_sup,L_edge,L_inv 55k/1.21.2k/22k 33 State Push-T state + 33 nuisance slots action vs. no-op reversed corr. (ρ=0.95ρ=0.95) ℒΔ,ℒedge,ℒinv,ℒctxL_ ,L_edge,L_inv,L_ctx 2020k/2.52.5k/2.52.5k 33 Vision Push-T 44 frozen slots, d=128d=128 action vs. no-op reversed pattern corr. (ρ=0.95ρ=0.95) ℒΔ,ℒinv,ℒctxL_ ,L_inv,L_ctx 1212k/22k/2.52.5k 33 LeWM Push-T 2242224^2 RGB, K=1K=1, d=192d=192 55-step macro vs. zero macro none; mass, visual (planning) ℒΔL_ 99k/11k pairs 33 CausalWorld 64264^2 RGB, K=1K=1, d=192d=192 first joint command vs. mirrored one visual, mechanism, composed ℒΔL_ (+ℒctx+L_ctx) 66k/11k/1.51.5k per split 33 model, 33 data Figure 4: Nuisances and shifts in each benchmark. In every benchmark the nuisance is correlated with the outcome during training and the correlation is reversed (or, for mechanism shifts, the physics is changed) at test time. The physics of the two members of a context twin is identical by construction. C.1 Synthetic SCMs Hard SCM (Q1, Q2). Seven slots of dimension 1616: four objects and three nuisance variables. The objects sit on a jittered ring, and a directed interaction links object i to object i+1mod4i+1 4. An action is a 44-vector: a contact point and a 2-D impulse. It never names its target. The contact point is the target’s position moved 18%18\% of the way towards the next object on the ring, plus Gaussian noise (s.d. 0.120.12); the impulse points roughly towards that next object. The first step after the action applies the impulse to the target only; the next two steps propagate velocity along the ring (strength 0.420.42, drag 0.900.90). A hidden context sets the nuisance values and biases the behaviour policy, so nuisances co-vary with the action (ρ=0.95ρ=0.95) in training; the test split reverses this. Both branches of a pair share the hidden context and the noise. Splits: 5,0005,000/1,0001,000/2,0002,000 (train/validation/test). Easy SCM. An earlier version with three objects and one nuisance (a “light”) driven by the same hidden context as the action policy; the test split reverses the correlation. It was retired because every Do-JEPA variant reached target accuracy 1.01.0 on it, so it could not tell the ingredients apart. Temporal SCM (App. E.2). Six objects on a directed ring and three nuisances (slot dimension 1818), with four propagation stages after a direct intervention. The edge loss sees only coarsened rollouts: κ=1κ=1 observes stages 0,1,2,3,4\0,1,2,3,4\, κ=2κ=2 0,2,4\0,2,4\, κ=3κ=3 0,3,4\0,3,4\ and κ=4κ=4 0,4\0,4\. The immediate factual/reference pair is available for every κ. Splits: 5,0005,000/1,2001,200/2,0002,000. C.2 State-based Push-T We use the gym-pusht simulator (Chi et al., 2023) with a 5-D state (agent position, block position and angle) and three synthetic nuisance slots. A pair branches the saved state under the factual action and a no-op; later rollout steps are identical. An early version restored only the body pose, not the collision and warm-start state, and replays differed by 1.61.6 units; we replaced it with a seeded reset plus identical context replay, which reproduces the branches exactly. Weak confound: nuisances correlated with the action in training and reversed at test. Strong confound: nuisances correlated with contact geometry at ρ=0.95ρ=0.95, plus input-only sensor corruption (agent noise 66 px, block noise 1818 px, angle noise 0.250.25 rad, dropout probability 0.150.15). Splits: 20,00020,000/2,5002,500/2,5002,500, rollout horizon 33. The multi-step dataset uses the strong-confound protocol with 1010-step action sequences. C.3 Vision Push-T 96×9696× 96 RGB frames. A nuisance pattern is rendered in the image border; in training it is correlated with contact and action geometry (ρ=0.95ρ=0.95, strength 0.450.45) and at test the correlation is reversed. The pattern never changes the physics, and the encoder saw both appearances during pretraining, so the test is a correlation shift, not an unseen style. Each sample has three history frames, a factual future and a no-op future from the exactly restored state. Frozen slots: C-JEPA’s Push-T VideoSAUR/DINOv2 encoder (44 slots ×128× 128). Splits: 12,00012,000/2,0002,000/2,5002,500. Before training any predictor, we checked that the slots contain the physics: linear probes decode agent position to 7.067.06 px, block position to 4.884.88 px and angle to 0.0260.026 rad (trivial baselines 8989 px, 113113 px, 1.611.61 rad), and the nuisance moves the slots 3.16×3.16× more than one step of physics. A home-made reconstruction slot autoencoder, tried first, gave probe errors of 7575 px and 83∘83 and was discarded. C.4 LeWM Push-T pairs and planning corpora The LeWM studies use the stable-worldmodel Push-T environment (Maes et al., 2026a) at 224×224224× 224. Five primitive controls form one 1010-D model action; the reference is five exact zero relative controls. Table 9 lists the corpora. Labels always come from measured outcomes (≥2≥ 2 px translation or ≥0.02≥ 0.02 rad rotation of the block), never from the proposal that generated the action. Table 9: Push-T intervention corpora for the LeWM studies. All use exact same-state branching with audited replay. “Resp.” is the fraction of responsive pairs. Corpus States n Resp. Action source Enriched (E) training episode replay 10,00010,000 50.0%50.0\% balanced toward/away proposal Outcome-unfiltered holdout episode replay 5,0005,000 5.9%5.9\% i.i.d. from expert action marginal Fresh outcome-unfiltered holdout episode replay 5,0005,000 6.5%6.5\% i.i.d. from expert action marginal Expert contiguous expert prefix 2,5002,500 50.8%50.8\% contiguous expert macro Expert shuffled expert prefix 2,5002,500 50.8%50.8\% same macro, order shuffled Expert i.i.d. marginal expert prefix 2,5002,500 36.2%36.2\% i.i.d. from expert marginal Natural (N) training expert prefix 10,00010,000 ≈51%≈51\% contiguous expert macro Reference contrast expert prefix 2,5002,500 52.7%52.7\% factual / other expert macro / zero The enriched corpus proposes actions towards or away from the block to reach a 50/5050/50 responsive split. Because this proposal is direction-biased, the corpus is stratified, not representative; App. F measures what this costs. The two “outcome-unfiltered” holdouts reject nothing on the basis of outcome, but each primitive action is drawn independently from the expert action marginal, which removes temporal structure. We call them outcome-unfiltered rather than natural for this reason. C.5 CausalWorld pushing The CausalWorld pushing task (Ahmed et al., 2020): a three-finger robot pushes a block. Observations are 64×6464× 64 RGB frames (upsampled to 224224 for the encoder) at steps 0,4,8,120,4,8,12 of a 1212-step window; four primitive 99-D joint commands form one 3636-D model action. A pair shares the snapshot, a warm-up and all commands except the first: the factual and reference first commands are mirrored about the current joint position. The label is the maximum difference in block displacement over the window: responsive if ≥2≥ 2 m, non-responsive if ≤0.5≤ 0.5 m, and rejected in between. Physical state is stored for evaluation only; models see RGB and actions. Splits and shifts: Split Visual shortcut Physics (mass, friction) Context–effect correlation Train / IID val present Space A +0.88+0.88 to +0.90+0.90 OOD visual reversed Space A ≈−0.90≈-0.90 OOD mechanism independent Space B (heavier) 0.00.0 OOD composed reversed Space B ≈−0.90≈-0.90 Block mass is in [0.015,0.044][0.015,0.044] in Space A and [0.046,0.099][0.046,0.099] in Space B (simulator units). The mechanism split changes the physics and also makes the visual context independent of the outcome, so it is not a physics-only shift. Splits: 6,0006,000 train, 1,0001,000 IID validation and 1,5001,500 per OOD split. The benchmark has a single manipulated block, so it does not test direct-support recovery. The label leak and the rebuild. The first version sampled responsive and non-responsive pairs separately: the collector accepted or rejected candidates depending on the requested class. Pre-intervention geometry then predicted the label: a classifier on sampling variables alone reached AUROC 0.8340.834, and the standardised mean differences of pre-contact distance were −1.01-1.01 to −0.86-0.86. Results obtained on that version are diagnostic only and are not used for any claim. The rebuilt version samples geometry and action first, runs both branches, and assigns the label from the outcome. It balances the classes by exact warm-up length, 11-m bins of pre-contact distance and fingertip distances, with a 12×12× candidate pool, and it assigns the context only after balancing. Gates: |SMD|≤0.10|SMD|≤ 0.10 for all five geometry variables and sampling-only AUROC ≤0.60≤ 0.60. The rebuilt benchmark has sampling-only AUROC 0.5000.500–0.5010.501 and all |SMD||SMD| near 10−610^-6. The three independent instances of Table 2 were generated with the same protocol from dataset seeds 1717, 2727 and 3737, and all passed the same gates. Appendix D Training and Evaluation Details Table 10 lists the settings used by the training scripts. All models use AdamW. Table 10: Training settings. “Sel.” is the checkpoint-selection rule; all selection uses IID validation data. lr: learning rate; wd: weight decay. Setting Settings Hard SCM Transformer predictor (width 6464, depth 22, 44 heads), history 33, 22 propagation steps (γ=0.55γ=0.55); 2525 epochs, batch 128128, lr 3×10−43× 10^-4, wd 10−410^-4; λΔ=5 _ =5, λsup=2 _sup=2, mask temperature 1.01.0 (sparse-mask rows without ℒsupL_sup) or 0.70.7 (with ℒsupL_sup, and all gated models), entropy regulariser 0.020.02 (a cardinality term, also 0.020.02, is zero under the softmax mask), no-op mask penalty 0.050.05; every model is trained and evaluated with one random history slot masked (masked-history loss 0.250.25); edge runs add λinv=5 _inv=5 (threshold 0.050.05), λedge=5 _edge=5, gate invariance 55, gate ℓ1 _1 0.020.02, onset threshold 0.080.08, gate temperature 0.40.4. Sel.: lowest IID prediction MSE. Seeds 77 (ablations), 7/17/277/17/27 (gated model). Easy SCM Transformer predictor (width 6464, depth 22, 44 heads); 2525 epochs, batch 128128, lr 3×10−43× 10^-4, wd 10−410^-4; λΔ=5 _ =5, λsup=2 _sup=2, mask temperature 0.70.7, entropy regulariser 0.020.02; 5,0005,000/1,0001,000/2,0002,000 samples. Sel.: lowest IID prediction MSE. Seeds 7/17/27/37/477/17/27/37/47. Temporal SCM Width 6464, depth 22, 44 propagation steps; 2525 epochs, batch 128128, lr 3×10−43× 10^-4, wd 10−410^-4; λΔ=5 _ =5, λsup=2 _sup=2, λinv=5 _inv=5, λedge=5 _edge=5. Sel.: lowest IID prediction MSE. Seeds 7/17/277/17/27. State Push-T Width 9696, depth 33, 44 heads; gate temperature 0.70.7; 3030 epochs, batch 256256, lr 3×10−43× 10^-4, wd 10−410^-4; reference-branch weight 0.50.5, λΔ=5 _ =5, reward head 11, λinv=5 _inv=5 (threshold 0.030.03), λedge=2 _edge=2, gate invariance 22, gate ℓ1 _1 0.010.01, λctx=0.25 _ctx=0.25 (context twins by shuffling nuisances within a batch; evaluated by sign flip). Sel.: IID prediction and effect error (context not used). Seeds 7/17/277/17/27. Vision Push-T Frozen slots; transformer predictor (width 256256, depth 66, 88 heads); 4040 epochs, batch 512512, lr 3×10−43× 10^-4, wd 10−510^-5; λΔ=4 _ =4, λinv=1 _inv=1 (soft weights, temperature 0.350.35), context-twin prediction weight 0.50.5, history loss 11, λctx=0.5 _ctx=0.5 (sensitivity run 0.250.25). Sel.: IID validation. Seeds 7/17/277/17/27. LeWM Push-T (Paired vs. Aug) Released LeWM checkpoint (ViT-tiny/14 encoder, 192192-D latent, history 33, 66-layer predictor); 1010 epochs, batch 6464, lr 10−510^-5, wd 10−310^-3, SIGReg 0.090.09, reference-branch weight 1.01.0, λΔ=0.5 _ =0.5, gradient clipping 1.01.0, 10%10\% validation split, expert replay in every update. Seeds 77 (development), 17/27/3717/27/37 (confirmation). Sel.: lowest validation paired-prediction loss, the same rule for Aug and Paired (no effect or planning metric). Planning adapters Rank-1616 bias-free adapter (α=16α=16) on the frozen LeWM predictor; 1010 epochs, batch 6464, lr 10−410^-4, wd 00; reference-branch weight 1.01.0, λΔ=0.5 _ =0.5 (Paired only), 55-step rollout preservation weight 1010. Sel.: lowest held-out paired prediction MSE among epochs ≥1≥ 1 whose 55-step rollout error is at most 1.2×1.2× the base model’s; effect and planning metrics are not used. Seeds 77, 17/27/3717/27/37 (enriched), 7/17/277/17/27 (natural). LeWM, full objective Official LeWM training recipe (lr 5×10−55× 10^-5, wd 10−310^-3, bf16, SIGReg 0.090.09, frameskip 55, warm-up cosine schedule), four branches flattened into one batch of 128128; α=1α=1, β=0.5β=0.5; update budget matched to standard LeWM. Seed 77. CausalWorld LeWM-class encoder and predictor, 192192-D latent, identical initialisation for all conditions; lr 5×10−55× 10^-5, wd 10−310^-3, SIGReg 0.090.09, bf16; batch 128128 (3232 pairs × four branches for paired conditions); 40,00040,000 updates with a cosine schedule (5,0005,000 in the first run, 1010k and 2020k in the budget study); validation every 250250 updates; α=1α=1; β=0.25β=0.25 for Do-JEPA. Sel.: lowest IID latent factual MSE; a frozen causal-aware selector was also tested. Model seeds 47/57/6747/57/67; model seed 107107 with dataset seeds 17/27/3717/27/37. Evaluation and aggregation. Metrics are computed on held-out splits after checkpoint selection. Seeds are aggregated by the mean; we report the standard deviation or the per-seed values. Trainers derive their train/validation split from the run seed, so effect metrics are always computed within seed; evaluating a model on another seed’s split would expose up to 90%90\% of its training data. Hardware. All experiments ran on NVIDIA A100 40 GB GPUs, one GPU per run. Appendix E Additional Results E.1 Synthetic SCMs Table 11: Easy SCM, reversed-correlation test split, mean over five seeds. Every mask-based variant reaches object-only target top-1 of 1.01.0. “Nuis. effect” is the RMS of the predicted effect on the nuisance slot. Mask with ℒΔL_ has the lowest effect MSE but 282×282× more predicted nuisance effect than the full model. Best in bold, second best underlined. Pred. MSE ↓ Effect MSE ↓ Nuis. effect ↓ Nuis. mask ↓ Mask (global action) 0.006120.00612 0.003080.00308 0.08750.0875 — Mask +ℒΔ+\,L_ 0.002800.00280 0.0000840.000084 0.01100.0110 — Sparse mask 0.004460.00446 0.002580.00258 0.006060.00606 0.02570.0257 Sparse mask +ℒΔ+\,L_ 0.00279¯ 0.00279 0.0002610.000261 0.00240¯ 0.00240 0.0112¯ 0.0112 Sparse mask +ℒΔ+ℒsup+\,L_ +L_sup 0.002760.00276 0.000232¯ 0.000232 0.0000390.000039 0.0001970.000197 Table 12: Where does the effect travel? Synthetic SCM, reversed-correlation test split; seed 7 except the last row (mean over seeds 7/17/27). Every row also uses ℒΔ+ℒsupL_ +L_sup and the effect part of ℒinvL_inv. Edge AUROC and the true-edge gate are scored against the test pairs’ onset labels. “Off-path gate”: mean gate on slot pairs that are not on the rollout’s propagation path (this includes correctly learned but unused ring edges); “nuis. in-gate”: mean gate into nuisance slots; “nuis. effect”: as in Table 1. Edge AUROC ↑ True-edge gate ↑ Off-path gate ↓ Nuis. in-gate ↓ Nuis. effect ↓ Direct-effect MSE ↓ Gates without ℒedgeL_edge 0.6240.624 0.9740.974 0.4650.465 1.0×10−31.0× 10^-3 5.6×10−55.6× 10^-5 0.01290.0129 +ℒedge+\,L_edge (onset labels) 0.9750.975 0.9550.955 0.0530.053 2.7×10−42.7× 10^-4 3.9×10−53.9× 10^-5 0.01340.0134 ++ gate term of ℒinvL_inv 0.9750.975 0.9470.947 0.0520.052 1.0×10−41.0× 10^-4 2.0×10−52.0× 10^-5 0.01350.0135 same, 3 seeds 0.9751±0.00020.9751±0.0002 0.9500.950 0.0510.051 8.9×10−58.9× 10^-5 1.9×10−51.9× 10^-5 0.01370.0137 Evaluation protocol. Our evaluators log object-only top-1; we reloaded the saved checkpoints of Table 1 and recomputed top-1 over all slots with the same test split and masking (object-only values are reproduced exactly). The sparse-mask rows put almost all mask mass on nuisances (1.0001.000), so their all-slot top-1 is 0.0000.000; the support-trained row and the three gated models score the same under both definitions (0.99950.9995–1.0001.000). Every row is evaluated with one random history slot masked, as in training. C-JEPA itself predicts from the full history at test time, and the action path of the mask-based rows reads the unmasked last frame, which Mask does not see. The sparse-mask rows without ℒsupL_sup use mask temperature 1.01.0 and the row with ℒsupL_sup uses 0.70.7; we did not run a temperature-matched control, so the jump in Table 1 reflects both changes. Easy SCM and gated models. Table 11 gives the easy SCM. Each added component removes about an order of magnitude of predicted nuisance effect; the support loss removes the last factor of 6161. In the hard SCM, the three seeds of the gated model give prediction MSE 0.00410.0041, direct-effect MSE 0.01370.0137, target F1F_1 0.9995±0.00030.9995± 0.0003, target mask 0.9940.994, descendant mask 0.0030.003 and nuisance in-gate 8.9×10−58.9× 10^-5. Structural recovery per seed (mean test gate thresholded at 0.50.5): precision == recall =1=1 on seeds 77, 1717 and 2727; mean ring-edge gate 0.9470.947, 0.9510.951, 0.9530.953; mean gate on other pairs 0.00530.0053, 0.00250.0025, 0.00180.0018; largest non-edge gate 0.0590.059. E.2 Temporal resolvability Table 13: Temporal SCM, reversed-correlation test split, mean over seeds 7/17/27. κ causal stages are hidden in one observed step. “Onset-label F1F_1” scores the training labels themselves as one-hop edges. The schedule-aware relation is reachability within the observation interval. κ 11 22 33 44 Direct-target F1F_1 0.9970.997 0.9970.997 0.9970.997 0.9970.997 Descendant-effect F1F_1 0.9470.947 0.8890.889 0.8890.889 0.8890.889 One-hop structural F1F_1 0.9990.999 0.5490.549 0.5020.502 0.3370.337 One-hop structural AUROC 1.0001.000 0.9100.910 0.8840.884 0.8430.843 Onset-label F1F_1 1.001.00 0.400.40 0.400.40 0.250.25 Schedule-aware F1F_1 0.9990.999 0.9350.935 0.9920.992 0.8940.894 Schedule-aware AUROC 1.0001.000 1.0001.000 1.0001.000 0.7710.771 Prediction MSE 0.01050.0105 0.01030.0103 0.01050.0105 0.01040.0104 Table 13 gives the full results. The implied reachability horizon of the learned graph matches the schedule’s effective horizon (1,3,3,4)(1,3,3,4) on every seed. At κ=4κ=4 the learned graph is too dense (edge density 1.01.0 against 0.80.8 for the schedule relation), which lowers its AUROC. The multi-timescale model has one gate head per schedule; coarse heads are tied to reachability of the one-hop head by a differentiable max–min path composition, G(κ)≈Reachhκ(G(1))G^(κ) _h_κ(G^(1)). It reaches one-hop F1=0.977F_1=0.977 at κ=1κ=1 and schedule F1F_1 0.955/0.977/0.9090.955/0.977/0.909 at κ=2/3/4κ=2/3/4 (3 seeds). A single shared gate trained on all schedules reaches one-hop F1≈0.509F_1≈ 0.509. Tying the heads improves cross-scale consistency by +0.083+0.083, +0.070+0.070 and +0.022+0.022 F1F_1 over independent heads, at a small one-hop cost (0.9770.977 vs. 0.9990.999). E.3 State-based Push-T Causal routing and gates. Table 14 gives the four models of Sec. 4.1. Table 14: State-based Push-T, reversed-correlation split, mean of seeds 7/17/27. “Agent pos.” is the agent-position MSE in px2; “nuis. effect” is the mean per-slot RMS of the predicted effect on nuisance slots; edge AUROC is scored against the test pairs’ onset labels. The agent is the known direct target in all rows. Arrows give the better direction. Best in bold, second best underlined. Pred. MSE ↓ Agent pos. ↓ Nuis. effect ↓ Edge AUROC ↑ Nuis. in-gate ↓ Obs (global action) 2.77×10−42.77× 10^-4 5.495.49 1.9×10−31.9× 10^-3 — — Global action, +ℒΔ+ℒinv+L_ +L_inv 2.78×10−42.78× 10^-4 3.163.16 4.8×10−44.8× 10^-4 — — Action routed to agent, no gates 2.75×10−4¯ 2.75× 10^-4 0.930.93 2.2×10−4¯ 2.2× 10^-4 0.5000.500 1.0001.000 Routed, ++ gates, ℒedgeL_edge, gate invariance 2.72×−2.72× 10^-4 0.95¯ 0.95 2.1×−2.1× 10^-7 0.9940.994 3.1×−3.1× 10^-5 Figure 5: Context weight in state Push-T, strong confound. Left: IID and OOD prediction error against λctx _ctx (seed 7); the circled point (both panels) is λctx=0.25 _ctx=0.25, chosen as the point on the IID–OOD trade-off curve closest to the ideal, so this seed’s OOD split informed the choice. Right: the context-induced block displacement falls monotonically with λctx _ctx. Context weight. Per seed, λctx=0.25 _ctx=0.25 against λctx=0 _ctx=0 changes the IID error by +7.0%+7.0\%, +4.2%+4.2\% and +3.5%+3.5\%, the OOD error by −59.0%-59.0\%, −43.9%-43.9\% and −55.2%-55.2\%, and the OOD/IID ratio from 3.693.69, 2.722.72, 2.952.95 to 1.411.41, 1.461.46, 1.281.28 (seeds 77, 1717, 2727). Edge AUROC is 0.9940.994 in every run. The first context run used λctx=5 _ctx=5, which closed 97%97\% of the gap at a 19%19\% IID cost. E.4 Vision Push-T Main comparison. Vision Push-T models share frozen VideoSAUR/DINOv2 slots (Zadaianchuk et al., 2023; Oquab et al., 2024) from C-JEPA’s Push-T encoder; a pattern rendered in the image border is correlated with contact geometry (ρ=0.95ρ=0.95) in training and reversed at test. We compare Mask, Aug and Do-JEPA (ℒΔ+ℒinv+ℒctxL_ +L_inv+L_ctx; λctx=0.5 _ctx=0.5 chosen on IID data; Table 15). Do-JEPA beats C-JEPA-style masking on every metric and every seed: factual error 39.0→27.239.0→ 27.2 px (30%30\% lower), responsive effect error 20.3→17.920.3→ 17.9 px (12%12\% lower), cosine 0.60→0.740.60→ 0.74, response AUROC 0.49→0.690.49→ 0.69 (masking is at chance), and context shift of predicted effects 18%18\% lower in latent space and 31%31\% lower in pixels. Against Aug, which sees the same branches, the result is mixed: Do-JEPA detects better whether the block will respond (AUROC +0.025+0.025, 3/33/3 seeds), and its predicted effects vary 20%20\% less with the context in latent space (3/33/3); but it has 9.3%9.3\% higher factual error and 6.5%6.5\% higher responsive effect error than Aug, and in pixels Aug’s predicted effects are less context-sensitive (12.612.6 vs. 13.713.7 px; 0/30/3 seeds favour Do-JEPA). Table 15: Vision Push-T, frozen VideoSAUR slots, reversed-pattern OOD split, mean± over seeds 7/17/27. Block effects are decoded to pixels by a context-balanced probe trained on IID data. “Latent ctx.” is the context shift of predicted effects in slot space; “Phys. ctx.” is the same shift decoded to pixels. Arrows give the better direction. Best in bold, second best underlined. Factual (px) ↓ Resp. effect (px) ↓ Cosine ↑ AUROC ↑ Latent ctx. ↓ Phys. ctx. (px) ↓ Mask (C-JEPA-style) 39.02±1.2439.02± 1.24 20.25±0.9620.25± 0.96 0.600±0.1080.600± 0.108 0.485±0.0200.485± 0.020 1.003±0.054¯ 1.003± 0.054 19.91±1.5019.91± 1.50 Aug 24.88±0.5824.88± 0.58 16.79±0.1116.79± 0.11 0.763±0.0100.763± 0.010 0.666±0.008¯ 0.666± 0.008 1.024±0.0141.024± 0.014 12.59±0.5112.59± 0.51 Do-JEPA 27.19±0.16¯ 27.19± 0.16 17.89±0.20¯ 17.89± 0.20 0.744±0.013¯ 0.744± 0.013 0.691±0.0080.691± 0.008 0.819±0.0190.819± 0.019 13.74±0.68¯ 13.74± 0.68 Table 16: Vision Push-T screen (seed 7, OOD). “Paired pred.” predicts both branches as separate targets (an Aug without context twins); “+ ctx. twins” adds context-swapped copies as data (Aug). Do-JEPA here uses λctx=2 _ctx=2. The true latent effect is itself context-sensitive (target context shift 2.132.13). Best in bold, second best underlined. Factual (px) ↓ Resp. eff. (px) ↓ Cos. ↑ AUROC ↑ Lat. ctx. ↓ Phys. ctx. (px) ↓ Object-centric JEPA, no mask 36.6836.68 21.1321.13 0.5600.560 0.4810.481 1.0211.021 20.4520.45 Mask 40.4140.41 21.3021.30 0.4790.479 0.4650.465 1.008¯ 1.008 21.2221.22 Paired pred. 33.2333.23 20.5120.51 0.6200.620 0.5290.529 1.0271.027 17.0717.07 Paired pred. ++ ctx. twins (Aug) 24.3124.31 16.8316.83 0.7560.756 0.672¯ 0.672 1.0091.009 12.3012.30 Paired 37.1537.15 20.6220.62 0.5970.597 0.5370.537 1.0661.066 17.4317.43 Do-JEPA (λctx=2 _ctx=2) 26.78¯ 26.78 18.68¯ 18.68 0.727¯ 0.727 0.7030.703 0.5150.515 13.71¯ 13.71 Figure 6: Context weight in vision Push-T (seed 7, OOD). Do-JEPA across λctx _ctx (orange) against C-JEPA-style masking (dashed) and Aug (dotted) on the same split. Do-JEPA beats masking on every metric at every weight, and a larger weight halves the latent context shift. The circled weight was selected on IID data as the smallest one with at least 20%20\% lower latent context shift and at most 5%5\% higher effect RMSE, 0.020.02 lower AUROC and 0.030.03 lower cosine; OOD values are shown. Screen and weight sweep. Table 16 gives the six-condition screen and Fig. 6 the weight sweep. The three-seed results (Table 15) also include a λctx=0.25 _ctx=0.25 run: factual 27.1727.17 px, responsive effect 17.5317.53 px, cosine 0.7510.751, AUROC 0.6920.692, latent context shift 0.9640.964, physical context shift 13.7613.76 px. E.5 LeWM Push-T: per-seed results Table 17: Effect loss on LeWM, pixel Push-T, per seed (Aug → Paired). Left: held-out pairs from the training distribution (half responsive). Right: fresh outcome-unfiltered holdout (5,0005,000 pairs, 323323 responsive). All metrics are latent, in each model’s own latent space; cosines are over all pairs. Factual columns give the relative change in factual MSE. Held-out training-distribution pairs Outcome-unfiltered holdout Seed Effect RMSE ↓ Cosine ↑ Factual MSE Effect RMSE ↓ Cosine ↑ Factual MSE 1717 0.2123→0.15070.2123→ 0.1507 0.669→0.7380.669→ 0.738 −40.9%-40.9\% 0.1031→0.09140.1031→ 0.0914 0.177→0.1490.177→ 0.149 −17.7%-17.7\% 2727 0.2118→0.15120.2118→ 0.1512 0.683→0.7440.683→ 0.744 −38.4%-38.4\% 0.1020→0.09060.1020→ 0.0906 0.169→0.1450.169→ 0.145 −17.3%-17.3\% 3737 0.2160→0.15650.2160→ 0.1565 0.663→0.7300.663→ 0.730 −39.9%-39.9\% 0.1020→0.09070.1020→ 0.0907 0.173→0.1450.173→ 0.145 −17.3%-17.3\% Mean change −28.4%-28.4\% +0.066+0.066 −39.7%-39.7\% −11.2%-11.2\% −0.027-0.027 −17.5%-17.5\% The outcome-unfiltered holdout was checked with a standalone evaluator that reproduces the original evaluation exactly (difference 0.00.0). On its responsive subset, the mean change is −0.5%-0.5\% in effect RMSE and −0.068-0.068 in cosine; the overall gain comes from non-responsive pairs (−17.3%-17.3\%). The single-seed development run gave −29.1%-29.1\% effect RMSE and +0.077+0.077 cosine. Physical read-out. The metrics above live in each model’s own latent space. To read them in physical units we fit, for each model, a ridge map from its true latent paired effect za−za∅z^a-z^a_ to the block displacement (px) on its own training pairs (ridge weight chosen on a held-out tenth of them) and apply it to the model’s predicted effect z^a−z^a∅ z^\,a- z^\,a_ . This post-hoc diagnostic uses privileged labels for evaluation only (Table 19). On held-out pairs from the training distribution, Paired lowers the responsive physical effect error on every seed (mean 7.0%7.0\%; 8.2%8.2\% over all pairs) and turns the responsive direction from wrong to right (cosine −0.19→+0.23-0.19→+0.23). On the 323323 responsive pairs of the outcome-unfiltered holdout it is 2.62.6–3.2%3.2\% worse (cosine 0.18→0.110.18→ 0.11). Absolute accuracy is low for both models: their responsive error is close to predicting no displacement, whereas decoding the true latent effects reaches 6.06.0–9.59.5 px (cosine 0.710.71–0.750.75), so the latent code holds more of the effect than either predictor uses. Training on natural action sequences. To test whether the gain follows the training intervention distribution, we trained Aug and Paired with the unchanged recipe on the natural corpus (10,00010,000 pairs with contiguous expert macros; Table 9) and evaluated all twelve models on the expert-contiguous arm of the natural sequence benchmark (2,5002,500 pairs whose episodes are excluded from the natural training corpus) and on the responsive pairs of the outcome-unfiltered holdout. The read-out is fitted per model on the natural corpus. The design was fixed in the protocol file before the runs (Table 18). Trained on natural sequences, Paired lowers the responsive physical effect error on the natural test set by 13.313.3–13.6%13.6\% and on the holdout by 5.35.3–5.7%5.7\%; trained on the enriched corpus it is 6%6\% and 44–5%5\% worse on the same sets. The natural-trained models are also far more accurate in absolute terms (responsive error 1010–1212 px against 19.719.7 px for predicting no displacement; cosine 0.800.80). The same pattern appears in the planning transfer matrix (App. F). Table 18: The LeWM Push-T gain follows the training intervention distribution. Paired vs. Aug trained with the same recipe on the enriched or the natural corpus; per-seed relative change (seeds 17/27/37) and mean responsive physical cosine (Aug → Paired). Latent effect RMSE is over all pairs, physical error over responsive pairs. Trained on Tested on Latent effect RMSE (%) Physical effect RMSE (%) Physical cosine Natural natural test (2,5002,500) −18.1/−18.4/−18.2-18.1/-18.4/-18.2 −13.6/−13.6/−13.3-13.6/-13.6/-13.3 0.754→0.7980.754→ 0.798 Natural unfiltered holdout (323323) −8.6/−8.1/−8.2-8.6/-8.1/-8.2 −5.7/−5.6/−5.3-5.7/-5.6/-5.3 0.441→0.5470.441→ 0.547 Enriched natural test (2,5002,500) −3.6/−3.1/−3.2-3.6/-3.1/-3.2 +5.8/+5.8/+6.1+5.8/+5.8/+6.1 0.354→0.2410.354→ 0.241 Enriched unfiltered holdout (323323) −1.0/−0.2/−0.3-1.0/-0.2/-0.3 +3.7/+5.3/+5.2+3.7/+5.3/+5.2 0.168→0.1010.168→ 0.101 Table 19: Physical read-out of the LeWM Push-T effect gain (Aug → Paired, per seed; responsive pairs; px). “Zero”: error of predicting no displacement; “decoded true”: the read-out applied to the true latent effects (Aug / Paired). Held-out training-distribution pairs Outcome-unfiltered holdout Seed Effect RMSE ↓ Cosine ↑ Zero Decoded true Effect RMSE ↓ Cosine ↑ 1717 11.75→11.0311.75→ 11.03 −0.188→0.240-0.188→ 0.240 10.9610.96 7.497.49 / 7.247.24 13.21→13.5613.21→ 13.56 0.206→0.1240.206→ 0.124 2727 11.73→10.6711.73→ 10.67 −0.198→0.237-0.198→ 0.237 10.6710.67 6.236.23 / 6.026.02 13.29→13.7113.29→ 13.71 0.177→0.1120.177→ 0.112 3737 13.78→13.0013.78→ 13.00 −0.170→0.212-0.170→ 0.212 12.6312.63 9.549.54 / 9.339.33 13.38→13.7713.38→ 13.77 0.160→0.0840.160→ 0.084 E.6 CausalWorld Table 20: CausalWorld fresh-seed confirmation (model seeds 47/57/67, 4040k updates, β=0.25β=0.25, frozen selector, OOD opened once). Relative changes of the candidate against the reference, mean over seeds. The rule for each split: at least one benefit (AUROC +0.02+0.02, cosine +0.05+0.05 or 10%10\% lower responsive effect error) with at most 10%10\% higher factual error, on at least 2/32/3 seeds. Hypothesis Split Resp. effect RMSE Cosine AUROC Factual RMSE Ctx. shift Seeds Result H1: Paired vs. Aug visual 11.3%11.3\% lower +0.056+0.056 −0.019-0.019 17.2%17.2\% higher — 1/31/3 not met mechanism 22.0%22.0\% lower +0.067+0.067 −0.022-0.022 15.1%15.1\% higher — 1/31/3 not met composed 24.3%24.3\% lower +0.059+0.059 −0.012-0.012 17.8%17.8\% higher — 1/31/3 not met H2: Do-JEPA vs. Paired visual 15.2%15.2\% higher −0.130-0.130 −0.019-0.019 33.3%33.3\% higher 47.2%47.2\% lower 0/30/3 not met mechanism 7.1%7.1\% higher −0.113-0.113 −0.021-0.021 31.7%31.7\% higher 46.1%46.1\% lower 0/30/3 not met composed 8.8%8.8\% higher −0.125-0.125 −0.031-0.031 32.3%32.3\% higher 46.6%46.6\% lower 0/30/3 not met H3: Do-JEPA vs. Aug visual 2.2%2.2\% higher −0.075-0.075 −0.038-0.038 56.2%56.2\% higher — 0/30/3 not met mechanism 16.5%16.5\% lower −0.046-0.046 −0.043-0.043 51.6%51.6\% higher — 0/30/3 not met composed 17.7%17.7\% lower −0.066-0.066 −0.043-0.043 55.8%55.8\% higher — 0/30/3 not met Training budget and a single-seed pass. On the rebuilt benchmark at 1010k updates, the IID rule (factual error at most 10%10\% higher, responsive effect error at most 5%5\% higher, cosine and AUROC at most 0.030.03 lower, context shift at least 20%20\% lower than Paired) selected β=0.25β=0.25. On seed 7 this model passed its OOD gate on all three splits against Aug, and Paired alone would also have passed (+0.10+0.10 to +0.11+0.11 cosine; 99–11%11\% lower responsive effect error), so the pass is attributable to pairing. This run was single-seed and not fully blind to OOD results, because OOD numbers for β=0β=0 and β=1β=1 were known. Across the checkpoints of longer runs, factual latent MSE and causal metrics were strongly anti-correlated (|Spearman|=0.93|Spearman|=0.93), so we froze a causal-aware selector for the fresh-seed test. Fresh-seed confirmation. Table 20 gives the full result. H1 reproduces the benefit of Table 2 on fresh model seeds (responsive effect error 1111–24%24\% lower, cosine +0.06+0.06) with a factual cost of 1515–18%18\%, just above the preregistered 10%10\% tolerance, so it is not confirmed (only seed 6767 passes); Table 21 shows that this cost is specific to training from scratch. The context term (H2, H3) lowers the latent context shift by 4646–47%47\% but costs accuracy, as discussed in Sec. 4.2. The frozen selector changed none of the nine checkpoints. Diagnosis. The audits of Sec. 4.2 use IID data only. The effect norm of the full model falls with depth relative to Paired: 23%23\% in the late encoder, 43%43\% after the projection and 48%48\% at the predictor output, where decoded physical error is 29%29\% higher (3/33/3 seeds). The pair and context gradients are not in conflict (cosine ≈0.69≈ 0.69), but the context gradient is 2.252.25 times larger and 58%58\% of it comes from the target side. Shrinkage is not the explanation: only 00–1.3%1.3\% of the suppressed discrepancy is radial, and rescaling effects made the gap larger. A predictor-side variant, which drops the target-side term, keeps the latent pair cosine at 0.940.94–0.950.95 while the physically weighted error rises by 76%76\%, 17%17\% and 12%12\%. The linear physical decoder has nine outputs; an SVD shows that its top two directions in the 192192-D latent space (not two coordinates) carry about 99.8%99.8\% of the physical error; the variant has 19.4%19.4\%, 9.6%9.6\% and 17.5%17.5\% more residual energy there (the audit criterion holds on 2/32/3 seeds). These diagnostics rely on decoders fitted with privileged labels. IID results on three benchmark instances. Before opening the OOD splits of Table 2, the IID validation results showed the same pattern on all three instances: responsive effect error 8.76→7.968.76→ 7.96 m, cosine 0.482→0.5370.482→ 0.537, context shift 2.38→0.792.38→ 0.79, factual error 15.07→22.8415.07→ 22.84 m, AUROC 0.880→0.8170.880→ 0.817 (Aug → Paired). Where the factual cost comes from (post-hoc diagnostic). The Push-T runs fine-tune a pretrained model; the CausalWorld runs train from scratch. We therefore started from each instance’s converged Aug checkpoint (the one locked for Table 2) and trained it for 2,0002,000 further updates as Aug or as Paired (α=1α=1), with learning rate 10−510^-5 held constant after 100100 warm-up updates, the same data and batch composition, and no checkpoint selection (the last checkpoint is evaluated). Probes were fitted as in App. C.5. The design was fixed in a protocol file before the runs; it was chosen after the results of Table 2. Table 21 gives the per-instance results: factual error stays within −4.4-4.4 to +2.0%+2.0\%, response AUROC is unchanged, and responsive effect error, cosine and latent context shift improve on every instance and split. We also retrained Paired from scratch with the full recipe (4040k updates, model seed 107107, IID factual-MSE selection) and the effect weight lowered to α=0.25α=0.25. Against the same Aug models, factual error rises by 3.7/11.5/15.8%3.7/11.5/15.8\% (IID; instances 17/27/37) instead of 52%52\%, responsive effect error falls by 10.2/6.5/10.5%10.2/6.5/10.5\% IID and 1010–15%15\% under the physics shifts, cosine rises by 0.020.02–0.090.09 and response AUROC falls by at most 0.0440.044. Against Paired with α=1α=1, factual error is 2525–30%30\% lower on every instance and split, IID responsive effect error is the same (−3.1-3.1 to +1.6%+1.6\%), and the latent context shift is about twice as large. The effect weight therefore sets the trade-off when the representation is trained from scratch (Table 3). Table 21: CausalWorld, fine-tuning a trained Aug model with the effect loss (2,0002,000 updates; Paired vs. Aug fine-tuned identically). Values are per benchmark instance (dataset seeds 17/27/37): relative changes in % for factual error, responsive effect error and latent context shift; differences for cosine and AUROC. ∗Also decorrelates the visual context. Split Factual RMSE (%) Resp. effect RMSE (%) Effect cosine Response AUROC Latent ctx. shift (%) IID −0.5/+1.7/−3.0-0.5/+1.7/-3.0 −6.1/−8.9/−9.5-6.1/-8.9/-9.5 +0.044/+0.051/+0.091+0.044/+0.051/+0.091 −0.006/−0.004/−0.001-0.006/-0.004/-0.001 −24.2/−24.7/−28.3-24.2/-24.7/-28.3 Visual −0.2/+2.0/−4.4-0.2/+2.0/-4.4 −7.5/−8.7/−10.6-7.5/-8.7/-10.6 +0.060/+0.050/+0.080+0.060/+0.050/+0.080 +0.009/+0.006/+0.010+0.009/+0.006/+0.010 −24.8/−24.4/−28.1-24.8/-24.4/-28.1 Mechanism∗ +0.1/+1.6/−4.0+0.1/+1.6/-4.0 −11.5/−12.5/−16.2-11.5/-12.5/-16.2 +0.042/+0.041/+0.085+0.042/+0.041/+0.085 +0.003/+0.006/+0.004+0.003/+0.006/+0.004 −25.7/−25.4/−28.4-25.7/-25.4/-28.4 Composed −0.8/+1.9/−4.2-0.8/+1.9/-4.2 −11.4/−12.7/−15.7-11.4/-12.7/-15.7 +0.042/+0.064/+0.079+0.042/+0.064/+0.079 +0.011/+0.015/+0.011+0.011/+0.015/+0.011 −25.5/−24.9/−28.5-25.5/-24.9/-28.5 Appendix F Planning Study Details Can paired supervision be added to a pretrained planner? Yes, with a centered adapter, but it does not improve planning; planning success never selects a checkpoint. Fine-tuning the released LeWM Push-T model (Maes et al., 2026b; Maes et al., 2026a) with ℒΔL_ lowers effect RMSE by 29.1%29.1\% relative to Aug but drops planning success from 94%94\% to 54%54\% (Aug: 58%58\%); the damage is in the predictor’s dynamics, not in the representation. A centered action-effect adapter (Eq. 11) adds gϕ(ha)−gϕ(ha←a∅)g_φ(h_a)-g_φ(h_a← a_ ) to the frozen predictor, so the no-op prediction cannot change. With a 55-step rollout-preservation loss and a checkpoint rule that caps rollout drift, it keeps 94%94\% success; these three parts were introduced together, so centering alone is not shown to suffice. Paired lowers effect RMSE by 5.25.2–5.8%5.8\% on four seeds, and at n=600n=600 episodes the adapted planners are equivalent to the base planner within ±3p± 3\,p. As in end-to-end training (App. E.5), the gain follows the training intervention distribution: adapters trained on natural sequences gain 2.47%2.47\% on natural actions, against 0.62%0.62\% for adapters trained on the enriched corpus. The rest of this section gives the details. Table 22: Ways of adding paired supervision to LeWM (seed 7, 5050-episode planning; effect metrics on each configuration’s own validation pairs, so compare within a row group only). “Adapted” says whether the selected checkpoint differs from the base model. Configuration Model Adapted Effect RMSE ↓ Cosine ↑ Planning ↑ Base LeWM — — — — 94%94\% Full fine-tuning Aug yes 0.21450.2145 0.6590.659 58%58\% Paired yes 0.15220.1522 0.7360.736 54%54\% Expert data only — yes 0.65880.6588 0.4280.428 %94\% Frozen representation Aug yes 0.29980.2998 0.8100.810 36%36\% Paired yes 0.28890.2889 0.8630.863 36%36\% Residual adapter Aug yes 0.65030.6503 0.4430.443 78%78\% Paired no 0.65800.6580 0.4410.441 (94%94\%) ++ preservation sweep Aug yes 0.63820.6382 0.4470.447 84%84\% Paired yes 0.58130.5813 0.5340.534 78%78\% Centered action-effect adapter Aug yes 0.60140.6014 0.5050.505 %94\% Paired yes 0.57010.5701 0.5420.542 %94\% The ladder. Table 22 gives every configuration. With full fine-tuning, the embedding cosine to the base model falls to 0.6890.689, and the Pearson correlation of squared latent distances to 0.5530.553. Freezing encoder and projector makes the embeddings bitwise identical (embedding MSE 00, cosine 11, distance correlation 11) and still gives 36%36\%. In the residual-adapter row, the Paired model’s selector returned epoch 00, the unchanged base model: its epoch-11 checkpoint missed the 1.20×1.20× preservation cap by 0.00060.0006. Its 94%94\% is the base model, not an adapted one; the next configuration forbids epoch 00. The Aug model of the preservation sweep has one-step prediction cosine 0.99940.9994 with the base model and plans 10p10\,p worse. zt−H+1:t,az_t-H+1:t,\ apredictortrunkoutputhead++Fadapt(z,a)F_adapt(z,a)zt−H+1:t,a←∅z_t-H+1:t,\ a_← predictortrunkgϕg_φgϕg_φ−-Fbase(z,a)F_base(z,a)δ _aha←∅h_a← sameweightsfrozenfrozensame frozen trunk;last action set to the no-optrainable, rank ≤16≤ 16δ=gϕ(ha)−gϕ(ha←∅)δ=g_φ(h_a)-g_φ(h_a← )δ=0δ=0 if a=a∅a=a_ Figure 7: Centered action-effect adapter (Eq. 11). The frozen predictor runs twice on the same history: with the action a, and with the last action replaced by the no-op. A trainable map gϕg_φ with shared weights is applied to both feature vectors, and their difference δ is added to the base prediction. δ is exactly zero for a=a∅a=a_ , whatever the parameters ϕφ. Properties of the adapter. With FbaseF_base the frozen pretrained predictor, hah_a its internal feature under action a, and ha←a∅h_a← a_ the feature with the same history and the last action replaced by the no-op, the centered action-effect adapter is Fadapt(z,a)=Fbase(z,a)+gϕ(ha)−gϕ(ha←a∅),F_adapt(z,a)=F_base(z,a)+g_φ(h_a)-g_φ(h_a← a_ ), (11) where gϕ(h)=Wϕhg_φ(h)=W_φh is bias-free, with Wϕ=αrBAW_φ= αrBA, A∈ℝ16×2048A ^16× 2048 and B∈ℝ192×16B ^192× 16 zero-initialised, so WϕW_φ starts at zero and has rank at most 1616. It is low-rank in the sense of Hu et al. (2022) and residual in the sense of Houlsby et al. (2019); the failure it avoids, the collapse of planning under fine-tuning, is a form of catastrophic forgetting (French, 1999; Kirkpatrick et al., 2017). It is trained with the paired prediction loss, a penalty on deviation from the base model’s own 55-step rollouts on expert data and, for Paired, ℒΔL_ ; a checkpoint is eligible only if its 55-step rollout error is at most 1.2×1.2× the base model’s (Table 10). Proposition 1 (Exact no-op invariance). For all z and all ϕφ, Fadapt(z,a∅)=Fbase(z,a∅)F_adapt(z,a_ )=F_base(z,a_ ). Proof. For a=a∅a=a_ the two branches of Eq. 11 receive the same input, so ha∅←a∅=ha∅h_a_ ← a_ =h_a_ and the adapter terms cancel. ∎ Proposition 2 (Linear, bounded correction). gϕ(ha)−gϕ(ha′)=Wϕ(ha−ha′)g_φ(h_a)-g_φ(h_a )=W_φ(h_a-h_a ), so ∥gϕ(ha)−gϕ(ha′)∥≤σmax(Wϕ)∥ha−ha′∥ g_φ(h_a)-g_φ(h_a ) ≤ _ (W_φ) h_a-h_a . Proposition 3 (Rollout drift). If the per-step correction has norm at most ε and the base transition is L-Lipschitz in its state, the k-step open-loop deviation is at most ε(Lk−1)/(L−1) (L^k-1)/(L-1) for L≠1L≠ 1 and kεk for L=1L=1. Proof. Let δj _j be the deviation after j steps. Then δj+1≤Lδj+ε _j+1≤ L _j+ with δ0=0 _0=0; unrolling gives δk≤ε∑j=0k−1Lj _k≤ _j=0^k-1L^j. ∎ Table 23: Numerical checks of the adapter properties (enriched-trained Paired adapter, seed 7, 256256 examples). Property Measured Check Prop. 1: no-op correction 0.00.0 (bitwise) pass Prop. 2: linearity max. error ≤2×10−7≤ 2× 10^-7 pass Prop. 2: operator bound holds for all pairs; mean tightness 0.0370.037 pass Prop. 3: sampled L L 0.7140.714 (lower bound, not certified) — Spectral / Frobenius norm of WϕW_φ 0.8570.857 / 1.2031.203 — Rank of WϕW_φ (192×2048192× 2048) ≤16≤ 16 by construction — Mean correction norm / mean latent norm 0.2820.282 / 13.913.9 — We introduced Prop. 3 expecting errors to grow geometrically. The measured drift is 0.280.28, 0.350.35, 0.380.38, 0.380.38 and 0.420.42 for k=1,…,5k=1,…,5, far below the bound evaluated with our Lipschitz estimate. That estimate (L^≈0.71 L≈ 0.71 for the enriched-trained adapter, 0.750.75 for the natural-trained one) is the mean over 88 batches of the largest finite-difference ratio found with 2424 random perturbations of the map from the 33-frame latent history to the next frame. It is a lower bound on the Lipschitz constant, not a certified value. Moreover, the rollout state is the whole 33-frame history, two frames of which are carried over unchanged at each step, so L^<1 L<1 does not show that rollouts contract. A consequence of Prop. 2: the no-op anchor cancels in any pairwise contrast, so the architecture can express do(a)do(a) versus do(a′)do(a ) without change. Whether the learned map transfers is tested below. Confirmation. Table 24 gives the three fresh seeds. All relative criteria of the protocol passed; its absolute criterion (≥90%≥ 90\%) was out of reach because the base model itself scores 83.0%83.0\% on these 200200 episodes. Table 24: Centered adapter, three fresh seeds. Effect RMSE on held-out enriched pairs; planning over 200200 episodes (base LeWM: 83.0%83.0\%). Seed 17 Seed 27 Seed 37 Effect RMSE, Aug 0.58350.5835 0.59560.5956 0.60430.6043 Effect RMSE, Paired 0.54960.5496 0.56400.5640 0.57260.5726 Relative reduction 5.80%5.80\% 5.31%5.31\% 5.25%5.25\% Rollout ratio, Paired 1.0321.032 1.0361.036 1.0331.033 No-op and representation unchanged yes yes yes Planning, Aug 81.5%81.5\% 84.5%84.5\% 84.5%84.5\% Planning, Paired 80.5%80.5\% 83.5%83.5\% 83.0%83.0\% Training distribution. Training fresh adapters on naturally occurring expert action sequences and crossing training with test distributions gives: Train E Train N Diagonal minus off-diagonal 95%95\% CI Test E 5.43%5.43\% 0.56%0.56\% 4.88p4.88\,p (9.8×9.8×) [4.66,5.09][4.66,5.09] Test N 0.62%0.62\% 2.47%2.47\% 1.85p1.85\,p (4.0×4.0×) [1.77,1.93][1.77,1.93] Both contrasts exclude zero: each adapter gains most on the distribution it was trained on. The natural-trained gain (2.47%2.47\%, CI [2.37,2.56][2.37,2.56]) is below the 5%5\% threshold fixed in the protocol but clearly above zero. Planning equivalence. At n=600n=600 episodes with per-episode pairing, the natural-trained Paired adapter (seed 7) reaches 87.7%87.7\% against 87.8%87.8\% for the base planner: a paired difference of −0.17p-0.17\,p, exact McNemar p=1.0p=1.0, and TOST equivalence at ±3p± 3\,p (90%90\% CI [−2.20,+1.87][-2.20,+1.87]). All six natural-trained adapters and the enriched-trained adapters are equivalent at ±3p± 3\,p (enriched Paired: −0.56p-0.56\,p, 90%90\% CI [−2.28,+1.17][-2.28,+1.17]). At n=200n=200 the same test passed at ±5p± 5\,p but not at ±3p± 3\,p. Appendix G Boundary Conditions and Diagnostic Findings Each item marks a condition under which the method, or a benchmark, does or does not apply. We give why it was considered, what we did, what we found, the lesson, and what it changes about our claims. Benchmarks outside the method’s assumptions. Three further benchmarks did not fit the method’s assumptions. CLEVRER has no actions or paired videos, and its counterfactual questions name objects that anonymous VideoSAUR or SAVi slots could not be matched to (soft IoU at most 0.0750.075), so named interventions could not be applied. In PHYRE, a support-trained model located the objects that respond to a placed ball much better than a matched control (support AUROC 0.8100.810 vs. 0.5830.583) but predicted their motion worse (responsive cosine −0.110-0.110 vs. +0.284+0.284): support identification is not effect prediction. In OGBench cube, exact MuJoCo restoration made the pairs exact, but the preliminary models were too undertrained for a conclusion. Propagation from anonymous pixel slots (single-seed IID diagnostic). Why: to test the propagation result beyond object-aligned variables. What we did: a preflight on new pixel Push-T pairs with frozen VideoSAUR slots and no object identities: 5,0005,000 pairs whose actions differ for 1010 physics steps and then share a 2020-step continuation. 79.2%79.2\% of pairs have a unique earliest responder with median onset margin 11 step and 2.3%2.3\% alignment ambiguity; each of the four slots is the earliest responder in about a quarter of the pairs, so roles are not tied to slot indices. We then trained the gated architecture with and without ℒedgeL_edge on seed 77 and evaluated it on the IID test split, with pass/fail rules fixed in advance. What we found: against the test pairs’ onset labels (anonymous slots have no ground-truth graph), edge AUROC rose from 0.8230.823 without ℒedgeL_edge to 1.0001.000 with it and the mean gate on other slot pairs fell from 0.3050.305 to 0.00020.0002, while effect RMSE changed by only 2.8%2.8\%, inside the 5%5\% margin. Direct-target top-1 was 1.01.0 on the 583583 evaluable cases. A context term added in the same study lowered the context shift by 1414–15%15\%, short of its 20%20\% target, in line with Sec. 4.2. Effect on claims: onset supervision trains gates on anonymous visual slots that reproduce onset order (one seed, IID data); propagation is claimed only with object-aligned variables.